acceptodds
Under review as a conference paper at ICLR 2027

PANORAMA: CROSS-MODEL KV SHARING VIA CONTEXTUAL RECOMPUTATION

11 views

Abstract

Real-world LLM workloads, such as multi-agent systems and model cascades that escalate requests to larger models, frequently switch a request between models while retaining the same context. Because KV caches are tied to model parame- ters and cannot be shared across models, each switch forces the target model to re-prefill the entire context that the source model has already processed, incur- ring substantial computation cost and increasingly delaying time to first token as context length grows. Existing cross-model KV translation methods avoid this re-prefill by mapping the source KV cache into an approximate target KV cache, but translation alone can substantially degrade accuracy. Selective KV recomputa- tion has been shown to repair context-inconsistent KV cache entries produced by the same model, but whether selective recomputation remains effective for a KV cache translated from a different model is unknown. Our key observation is that even when a translated KV cache is insufficient for direct decoding, the translated cache can still provide useful context for target-model recomputation. Based on this observation, we propose PANORAMA, a cross-model KV sharing method that recovers high-quality target inference from approximate cross-model KV with- out full target re-prefill. PANORAMA contextually recomputes a small subset of important target positions over the full KV cache produced by the cross-model mapper, allowing target-model computation to exploit the translated context. To provide the full translated context efficiently, PANORAMA factorizes the mapper into a shared source projection and layer-specific target reconstruction, transfer- ring only a compact latent per prompt position. PANORAMA further refines the mapper under the repaired mixed cache used at inference, removing the training– inference discrepancy between translation-only mapper fitting and mixed-cache decoding. Across model pairs and tasks, PANORAMA achieves the highest ac- curacy among prior cross-model handoff methods on nearly all benchmarks at comparable handoff cost. Moreover, in the Qwen3 8B-to-32B switching scenario, our vLLM implementation of PANORAMA reduces handoff latency by 5.84×over native re-prefill at 16K tokens with minimal accuracy loss.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
large language model serving, KV cache transfer, prefill reuse, cross-model serving, selective recomputation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rPZaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PANORAMA, a method for sharing KV caches when switching between language models processing the same context. It maps the source model’s KV cache into the target model’s representation and selectively recomputes target positions while allowing them to attend to the translated context. The method combines a factorized mapper for compact latent transfer with repair-aware refinement that trains the mapper using the mixed cache employed at inference. Evaluation covers Qwen3 and GLM-4 model pairs on LongBench, mathematical reasoning, and commonsense reasoning tasks, alongside component ablations and a vLLM implementation. For Qwen3-8B→32B with 10% recomputation, PANORAMA obtains a LongBench-16 score of 46.82 versus 47.99 for full target prefill. At 16K tokens, measured handoff latency decreases from 1,973.9 ms to 338.1 ms, a 5.84× speedup.

Critical strengths
  1. The paper identifies a useful role for translated KV beyond direct decoding. PANORAMA shows that an approximate translated cache can provide useful context for selective target-model recomputation, even when decoding directly from that cache performs poorly. The contextual-versus-isolated recomputation ablation supports this observation, providing an empirical basis for combining cache translation with selective recomputation.

  2. The design addresses both computation and communication costs. Selective recomputation reduces target-model computation, while the shared latent representation reduces the transferred KV-state payload. In the Qwen3-8B→32B setting, the design reduces this payload by 36× relative to dense source-KV transfer and achieves a 5.84× speedup over full target prefill in the measured 16K-token handoff setting. These results demonstrate the practical value of the combined design.

  3. The evaluation connects component-level evidence with downstream performance. Experiments cover two model families and multiple task categories, with ablations examining contextual recomputation, mapper refinement, and latent rank. On Qwen3-8B→32B, 10% recomputation yields LongBench-16 and GSM8K scores close to full target prefill. Together, these evaluations support the method’s quality–cost trade-off and help explain the contributions of its components.

Critical weaknesses
  1. The headline handoff speedup excludes online selection overhead. Section 4.1 and Appendix E specify that the handoff measurements use precomputed selection masks and exclude online selection and source look-ahead. The 5.84× speedup therefore characterizes the prepared-mask handoff stage. Although this measurement isolates transfer and target-side processing, the net latency benefit when online selection or source look-ahead lies on the critical path remains less clear.

  2. The additional benefit of the repair-aware refinement design is modest in the available ablation. Table 2(b) shows that refinement on the mapped cache already achieves 90.52% on GSM8K and 46.13 on LB-16, compared with 90.90% and 46.82 for RAR. Thus, much of the improvement over no refinement is recovered without using the recomputed mixed cache during training, as Section 4.3 also acknowledges. The results support mapper refinement, but provide more limited evidence for the additional contribution of the specifically repair-aware cache composition.

  3. Quality preservation varies substantially across model pairs and tasks. The near-native results on Qwen3-8B→32B do not consistently extend to the second model family. On GLM-4-9B→32B, GSM8K accuracy is 81.88%, compared with 92.87% for native prefill. On multi-document QA, PANORAMA scores 42.26, below SpecPrefill’s 47.09 and SpecPC’s 51.10. These acknowledged results show that PANORAMA can underperform selective-prefill baselines in some settings, limiting how broadly its near-native quality results can be generalized.

Questions for authors
  1. Could you clarify the online selection and source look-ahead costs, and whether these operations can overlap with other processing in the intended deployment? An estimate of their contribution to the switching latency would help interpret the practical speedup.

  2. Table 2(b) shows that refinement on the mapped cache recovers much of RAR’s improvement. Based on the available experiments, how consistently does training with the recomputed mixed cache provide additional gains? This would help assess the specific contribution of the repair-aware design.

  3. Code availability. I could not find an anonymous code repository link or a code attachment. Could you provide anonymized code and scripts to reproduce the reported results?

8 comments
AuthorsOpenReview ↗

[1/6] Response to Reviewer rPZa (W1)

W1. The headline handoff speedup excludes online selection overhead. Section 4.1 and Appendix E specify that the handoff measurements use precomputed selection masks and exclude online selection and source look-ahead. The 5.84× speedup therefore characterizes the prepared-mask handoff stage. Although this measurement isolates transfer and target-side processing, the net latency benefit when online selection or source look-ahead lies on the critical path remains less clear.

We are grateful for the careful attention to the latency measurements. In our original evaluation, we measured the handoff latency with precomputed selection masks, excluding source look-ahead and online selection, to isolate the cost of transferring and repairing the target KV cache. The reported 5.84× speedup therefore characterizes this handoff stage rather than the complete target-switching latency. With online selection and eight look-ahead tokens included, PANORAMA remains 4.79× faster than Native Prefill on the same 16K-token prompt. On the 400 scored LongBench requests, PANORAMA is 4.52× faster than Native Prefill and remains comparable in accuracy to Native Prefill. We detail each measurement below.


Additional Experiments: Online Selection and Benchmark-Matched Latency

We agree that look-ahead and online selection contribute to the actual switching latency.

To address this concern, we conducted additional experiments including both costs. That is, the latency runs from the switch to the target model to the first output token. We measured latency at the average prompt lengths of LB-16 (10K tokens) and GSM8K (1K tokens). Since look-ahead introduces additional latency, we also swept the look-ahead length instead of using only the eight-token setting. We have updated Figure 4 and Table 10 (Appendix E.2) in the revised manuscript with these measurements, and Table R3.1 below lists the key points.

Table R3.1. Accuracy and handoff latency including online selection. All measurements use Qwen3-8B→32B with a 10% recomputation budget. Each cell reports accuracy and handoff latency, and speedups over Native Prefill are in parentheses.

BenchmarkNative PrefillLook-aheadSpecPrefillPANORAMA (Ours)
LB-16 (10K)47.99 / 1,129 ms139.80 / 126 ms (9.0×)45.57 / 182 ms (6.2×)
442.20 / 149 ms (7.6×)46.40 / 191 ms (5.9×)
843.05 / 180 ms (6.3×)46.90 / 225 ms (5.0×)
GSM8K (1K)91.81% / 105 ms144.35% / 45 ms (2.3×)88.17% / 50 ms (2.1×)
449.66% / 68 ms (1.5×)91.05% / 71 ms (1.5×)
850.27% / 97 ms (1.1×)90.52% / 100 ms (1.0×)

The results show that PANORAMA retains higher accuracy than SpecPrefill at the same look-ahead and recomputation budget. The lower latency of SpecPrefill comes at a large loss in accuracy, and at a similar latency, PANORAMA still scores higher. For instance, on LB-16, PANORAMA with one look-ahead token runs in 182 ms, as fast as SpecPrefill with eight tokens at 180 ms, but scores 2.52 points higher. Figure 4 in the revised manuscript confirms this trend across recomputation budgets, where PANORAMA achieves the highest accuracy among the selective methods on long prompts. On short GSM8K prompts, the fixed cost of online selection becomes the main part of the latency. Four look-ahead tokens achieve 91.05% accuracy at 1.5× speedup, whereas eight tokens offer little speedup. With one look-ahead token, PANORAMA is 2.1× faster than Native Prefill, faster than SpecPrefill with four or eight tokens, and scores 37.90 points above the best SpecPrefill result. We therefore recommend one look-ahead token for short prompts or easy tasks and four or more tokens for long prompts or difficult tasks.


Additional Experiments: Latency and Accuracy on the Same Requests

To further verify the trade-off on actual workloads, we timed and scored the same 400 LongBench requests, preserving each request's original prompt length and using eight look-ahead tokens.

Table R3.2. Handoff latency and accuracy on the same 400 LongBench requests. All measurements use Qwen3-8B→32B with a 10% recomputation budget, and speedups over Native Prefill are in parentheses.

MethodHandoff latency (ms) ↓LB-16 ↑
Native Prefill1,215 (1.00×)48.96
SpecPrefill251 (4.83×)44.00
PANORAMA (Ours)269 (4.52×)47.55

On identical requests, PANORAMA achieves 4.52× faster target switching while remaining within 1.41 LB-16 points of Native Prefill. SpecPrefill is slightly faster, but PANORAMA scores 3.55 points higher at a similar latency. This demonstrates that the latency–accuracy trade-off persists on the evaluated LongBench requests even with online selection and source look-ahead on the critical path. These same-request results are reflected in Table 11 of Appendix E.3 of the revised manuscript.

AuthorsOpenReview ↗

[2/6] Response to Reviewer rPZa (Q1: Online Selection Cost)

Q1. Could you clarify the online selection and source look-ahead costs, and whether these operations can overlap with other processing in the intended deployment? An estimate of their contribution to the switching latency would help interpret the practical speedup.

We appreciate this practical question. We address the three parts in turn.

(1) Cost

As Table R3.3 shows, with eight look-ahead steps, online selection takes 62–90 ms for prompts ranging from 4K to 32K tokens. Most of this time comes from the source model's look-ahead decoding, while the scoring pass takes much less time. For example, at 10K tokens, look-ahead decoding takes 61.0 ms out of the total selection cost of 67.9 ms, accounting for about 90%. The cost of look-ahead decoding depends on the number of tokens generated, so fewer look-ahead steps further reduce the selection cost, as the look-ahead sweep in Table R3.1 shows.

Table R3.3. Stage-wise handoff latency with online selection on Qwen3-8B→32B with a 10% recomputation budget and eight look-ahead tokens. The total is the sum of the separately measured stages, and speedups are over Native Prefill.

PromptNative Prefill (ms)Prepared-mask handoff (ms)Look-ahead (ms)Scoring (ms)Total (ms)Speedup without selectionSpeedup with selectionSelection share
4K408.573.558.23.4135.15.56×3.02×46%
10K1,127.2178.461.06.9246.36.32×4.58×28%
16K1,973.9338.163.410.2411.75.84×4.79×18%
32K4,889.5738.269.919.7827.86.62×5.91×11%
AuthorsOpenReview ↗

[3/6] Response to Reviewer rPZa (Q1: Overlapping Selection with Computation)

Q1. Could you clarify the online selection and source look-ahead costs, and whether these operations can overlap with other processing in the intended deployment? An estimate of their contribution to the switching latency would help interpret the practical speedup.

(2) Overlap

In the intended deployments, both operations can overlap with the processing that precedes a target switch. Whether the two costs belong in the switching latency depends on what happens right before the switch. If a switch request arrived without any preceding step, both costs would lie on the critical path, as in Table R3.3. In practice, however, a switch to another model follows a decision step:

  • Human users read the answer of the smaller model and decide whether to escalate the request.
  • Model cascades escalate after the smaller model answers, when a scorer rejects the answer (FrugalGPT), when the smaller model fails self-verification and a router decides (AutoMix), or when the answers of the smaller model disagree (Yue et al., 2024).
  • Multi-agent orchestrators or selector LLMs read the history and choose the next agent (e.g., AutoGen, Magentic-One, Google ADK). In a deployed system, an LLM routing classifier alone takes 344–378 ms per routing decision (Shu et al., 2024).
  • Even in direct handoffs, the source agent first generates its reasoning and the handoff call (e.g., OpenAI Agents SDK, AutoGen Swarm).

Predefined workflows, which 16 of 20 deployed case studies use (Pan et al., 2025), even fix the next model in advance. In every case, the decision step takes place before the target model starts, and since look-ahead and selection run only on the source GPU and take 62–90 ms, they can finish within this step.

One may ask whether the target model could use the same decision step to run Native Prefill in advance. Between steps of the same model, an existing KV cache can simply be reloaded (KVFlow; CachedAttention), but a different target model has no cache to reload, so preparing the target requires computation on the target itself. Moreover, the decision step precedes the decision, so any preparation is wasted if no switch occurs. Table R3.4 compares three options for the decision step: (A) look-ahead and selection on the source, as in PANORAMA, (B) option A followed by recomputing the selected 10% on the target, and (C) full prefill on the target.

Table R3.4. Options for preparing a handoff during the decision step on Qwen3-8B→32B at 10K tokens. Preparation times are from Table R3.3.

A. Look-ahead + selection (PANORAMA)B. A + 10% recomputation on targetC. Full prefill on target
GPUs usedSourceSource + targetTarget
Target model loaded and runNot requiredRequiredRequired
Preparation time67.9 ms67.9 ms + target recomputation1,127.2 ms
Target GPU memory heldLatent only (4 KiB/token)Full mapped cacheFull target cache
Wasted if no switch occursSource compute and latent transferSource and target compute, target memoryTarget compute and memory
Other requests on the target GPUUnaffectedDelayed during preparationDelayed during preparation
Fits within a 344–378 ms routing decisionYesDepends on prompt lengthNo

Option A needs no target model, holds only the latent on the target, and leaves other requests on the target GPU unaffected, so it can be prepared before every potential switch, and little is wasted when no switch follows. Options B and C occupy the target GPU and its memory during the decision step and waste both whenever no switch occurs, and option C takes longer than the decision step itself. When a decision step precedes the switch, the switching latency is therefore the prepared-mask handoff of the 5.84× setting.

Even when no decision step precedes the switch, look-ahead overlaps with target-side work within the handoff. The source and target models run on separate GPUs, and the target model needs the selection mask only when it recomputes the selected tokens. Therefore, while the source model generates look-ahead tokens, the target model can receive the compact latent representation and rebuild the mapped KV cache. The length-matched measurement in Table R3.1 runs look-ahead concurrently in this way and gives 225 ms at 10K tokens, 21 ms below the serial sum of 246.3 ms in Table R3.3.

Reviewer_v9jMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PANORAMA translates source-model KV into target-model KV, then repairs selected positions using the translated context. A shared latent projection reduces transfer size, and output-based refinement improves the resulting mixed cache.

Critical strengths
  • The contextual-recomputation ablation supports the main insight that reading translated context raises GSM8K accuracy from 61.64% to 85.06% compared with isolated recomputation.
  • The compact mapper offers a useful trade-off: 36× smaller transferred state, with Qwen LongBench scores close to native prefill.
Critical weaknesses
  • The 5.84× latency gain excludes online selection and source lookahead, although inference requires eight lookahead steps on Qwen. Latency is also measured on one 16K prompt, separately from quality benchmarks, so Figure 4 does not establish the same trade-off on actual benchmark requests.
  • Near-native quality does not extend to GLM: GSM8K falls from 92.87% to 81.88%, and multi-document QA falls from 50.75 to 42.26. On the latter, PANORAMA also trails SpecPC’s 51.10 (Table 1).
  • Repair-Aware Refinement adds only 0.38 GSM8K points over refinement on the mapped cache. Since Ridge receives no equivalent output-based refinement, the experiments do not fully separate the benefit of the proposed training cache from additional training.
Questions for authors

Please see the weaknesses.

6 comments
AuthorsOpenReview ↗

[1/3] Response to Reviewer v9jM (W1)

W1. The 5.84× latency gain excludes online selection and source lookahead, although inference requires eight lookahead steps on Qwen. Latency is also measured on one 16K prompt, separately from quality benchmarks, so Figure 4 does not establish the same trade-off on actual benchmark requests.

We sincerely thank the reviewer for this insightful comment on how the latency gain holds in actual deployment. We apologize that the submitted version left the cost of online selection implicit. In our original evaluation, we measured the handoff latency with precomputed selection masks, excluding source look-ahead and online selection, to isolate the cost of transferring and repairing the target KV cache. The reported 5.84× speedup therefore characterizes this handoff stage rather than the complete target-switching latency. With online selection and eight look-ahead tokens included, PANORAMA remains 4.79× faster than Native Prefill on the same 16K-token prompt. On the 400 scored LongBench requests, PANORAMA is 4.52× faster than Native Prefill and remains comparable in accuracy to Native Prefill. We detail each measurement below.


Additional Experiments: Online Selection and Benchmark-Matched Latency

We agree that look-ahead and online selection contribute to the actual switching latency, and that a single 16K prompt does not establish the accuracy–latency trade-off on benchmark workloads.

To address these concerns, we conducted additional experiments including both costs. In other words, the latency runs from the switch to the target model to the first output token. We measured latency at the average prompt lengths of LB-16 (10K tokens) and GSM8K (1K tokens). Since look-ahead introduces additional latency, we also swept the look-ahead length rather than relying only on the original eight-token setting. We have updated Figure 4, and Table 10 in the revised manuscript with these measurements, and Table R2.1 below lists the key points.

Table R2.1. Accuracy and handoff latency including online selection. All measurements use Qwen3-8B→32B with a 10% recomputation budget. Each cell reports accuracy and handoff latency, and speedups over Native Prefill are in parentheses.

BenchmarkNative PrefillLook-aheadSpecPrefillPANORAMA (Ours)
LB-16 (10K)47.99 / 1,129 ms139.80 / 126 ms (9.0×)45.57 / 182 ms (6.2×)
442.20 / 149 ms (7.6×)46.40 / 191 ms (5.9×)
843.05 / 180 ms (6.3×)46.90 / 225 ms (5.0×)
GSM8K (1K)91.81% / 105 ms144.35% / 45 ms (2.3×)88.17% / 50 ms (2.1×)
449.66% / 68 ms (1.5×)91.05% / 71 ms (1.5×)
850.27% / 97 ms (1.1×)90.52% / 100 ms (1.0×)

The results show that PANORAMA retains higher accuracy than SpecPrefill at the same look-ahead and recomputation budget. The lower latency of SpecPrefill comes at a large loss in accuracy, and at a similar latency, PANORAMA still scores higher. For instance, on LB-16, PANORAMA with one look-ahead token runs in 182 ms, as fast as SpecPrefill with eight tokens at 180 ms, but scores 2.52 points higher. Figure 4 in the revised manuscript confirms this trend across recomputation budgets, where PANORAMA achieves the highest accuracy among the selective methods on long prompts. On short GSM8K prompts, the fixed cost of online selection becomes the main part of the latency. Four look-ahead tokens achieve 91.05% accuracy at 1.5× speedup, whereas eight tokens offer little speedup. With one look-ahead token, PANORAMA is 2.1× faster than Native Prefill, faster than SpecPrefill with four or eight tokens, and scores 37.90 points above the best SpecPrefill result. We therefore recommend one look-ahead token for short prompts or easy tasks and four or more tokens for long prompts or difficult tasks. The detailed accuracy–latency trade-off of PANORAMA against the other handoff methods, across all recomputation budgets and look-ahead lengths, can be found in Figure 4 and in Table 10 of Appendix E.2 of the revised manuscript.


Additional Experiments: Latency and Accuracy on the Same Requests

To further verify the trade-off on actual workloads, we timed and scored the same 400 LongBench requests, preserving each request's original prompt length and using eight look-ahead tokens.

Table R2.2. Handoff latency and accuracy on the same 400 LongBench requests. All measurements use Qwen3-8B→32B with a 10% recomputation budget, and speedups over Native Prefill are in parentheses.

MethodHandoff latency (ms) ↓LB-16 ↑
Native Prefill1,215 (1.00×)48.96
SpecPrefill251 (4.83×)44.00
PANORAMA (Ours)269 (4.52×)47.55

On identical requests, PANORAMA achieves 4.52× faster target switching while remaining within 1.41 LB-16 points of Native Prefill. SpecPrefill is slightly faster, but PANORAMA scores 3.55 points higher at a similar latency. This demonstrates that the latency–accuracy trade-off persists on the evaluated LongBench requests, beyond the original fixed-length, prepared-mask measurement. These same-request results are reflected in Table 11 of Appendix E.3 of the revised manuscript.

We have incorporated the benchmark-matched results, look-ahead sweep, and same-request evaluation into the revised manuscript and Appendices E.1–E.3.

AuthorsOpenReview ↗

[2/3] Response to Reviewer v9jM (W2)

W2. Near-native quality does not extend to GLM: GSM8K falls from 92.87% to 81.88%, and multi-document QA falls from 50.75 to 42.26. On the latter, PANORAMA also trails SpecPC’s 51.10 (Table 1).

We are very grateful for this constructive feedback, and we agree that the GLM-4 results deserved a closer look. The larger gap in the submitted GLM-4 results reflects an under-trained mapper configuration rather than a limitation of PANORAMA. In the submitted setting, the mapper biases shared the learning rate of the other mapper parameters and were trained for only one epoch, which was sufficient for Qwen3 but not for GLM-4. With a decoupled bias learning rate and longer training, PANORAMA more than halves the gap to Native Prefill on LB-16, GSM8K, and multi-document QA, outperforms both selective-prefill baselines on LB-16 and GSM8K, and moves ahead of SpecPrefill on multi-document QA. We detail these results below.


Additional Experiments: GLM-4 Quality with Revised Training

We revisited the mapper training configuration without changing the architecture, transfer payload, or inference procedure of PANORAMA. Specifically, we decoupled the bias learning rate from the mapper learning rate and trained the mapper on 3.87M tokens, eight times the 0.47M tokens of the submitted one-epoch setting. As Table R2.3 shows, these two changes improve PANORAMA on GLM-4 on LB-16, GSM8K, and multi-document QA.

Table R2.3. Accuracy on GLM-4-9B→32B with a 10% recomputation budget. The revised PANORAMA uses a decoupled bias learning rate of 2×10⁻³ and 3.87M training tokens, and Δ denotes the change from the submitted PANORAMA results.

MethodLB-16 ↑GSM8K (%) ↑Multi-doc QA ↑
Native Prefill49.1192.8750.75
SpecPrefill43.1256.4147.09
SpecPC43.3543.5251.10
PANORAMA (Ours), submitted44.3981.8842.26
PANORAMA (Ours), revised47.05 (Δ +2.66)88.02 (Δ +6.14)47.99 (Δ +5.73)

The gap on GLM-4 is wider than on Qwen3 because translation itself is weaker on this pair, as dense Ridge without recomputation reaches only 26.69% on GSM8K for GLM-4, compared with 72.40% for Qwen3 in Table 1, so less of the prompt survives as usable context. When translation is this weak, the mapped cache held as decoder state hurts most on multi-document QA, the category that depends most on exact evidence positions. This accuracy gap, however, closes with further refinement. With revised training, PANORAMA reduces the gap to Native Prefill from 10.99 to 4.85 points on GSM8K and from 8.49 to 2.76 points on multi-document QA. Although PANORAMA remains below SpecPC on multi-document QA (47.99 vs 51.10), the revised model recovers most of the original gap to Native Prefill (47.99 vs 50.75).

Reviewer_bXLLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Panorama, a cross-model KV sharing method that produces high-quality inferences from the target model based on an approximate KV cache computed from the KV cache on the source model. This method does not need re-prefill on the target model; instead, it recomputes only a small subset of important target positions using the approximate KV cache as context. The author's key observation is that even though the approximate cache translated from the source cache may be insufficient for direct encoding, it can serve as a useful context for target model computation. The experiments show that Panorama achieved similar accuracy performance to Native Prefill at a lower latency.

Critical strengths

S1. Panorama addresses the cross-model scaling problem by translating the source KV-cache to an approximate KV-cache on the target model, and uses this approximate KV-cache as context for the computation on the target without re-prilling the full prompt. It achieves a good accuracy with a short latency.

Critical weaknesses

W1. Tables and figures have some presentation problems. Table 1. Native Prefill has the highest performance on several benchmarks, but its numbers are not marked bold. Figure 4. the legend keys marked for Rige, CacheBridge are missing from the figures. Figure 5. The subfigure on the left has a dark bar unlabeled, and a legend key labeled as "Native Prefill" does not appear in the subfigure on the right.

Questions for authors

Q1. Table 1 does not mark Native Prefill numbers as bold, even though the Native Perefill metho produce the highest accuracy among all methods under stdy. Why? Is it Native Pefill does not count as a handoff method? Q2. Figure 4. The Ridge and CacheBridge legends do not appear in the figure even though their legend keys are on the top of the figure. Why? Q3. Figure 5. Native Prefill's legend key is a dashed line, but it doesn't appear in the subfigure on the left. Why? What's the dark bars on the left of all four bars in each batch size group.

2 comments
AuthorsOpenReview ↗

Response to Reviewer bXLL (Q1–Q3)

Q1. Table 1 does not mark Native Prefill numbers as bold, even though the Native Perefill method produce the highest accuracy among all methods under stdy. Why? Is it Native Pefill does not count as a handoff method?

We thank the reviewer for pointing out this ambiguity. As the reviewer correctly notes, Native Prefill does not count as a handoff method. Instead, we use Native Prefill as the quality reference: the target model recomputes the entire prompt without reusing the source KV cache. We intended boldface to indicate the best handoff result, but did not explain this convention clearly. We have now separated Native Prefill from the handoff methods and clarified the boldface convention in Tables 1 and 14 of the revised manuscript.

Q2. Figure 4. The Ridge and CacheBridge legends do not appear in the figure even though their legend keys are on the top of the figure. Why?

We apologize for the confusion in Figure 4. Ridge and CacheBridge translate the full source KV cache into a target KV cache, and the target model decodes from the translated cache without recomputing any position. Ridge and CacheBridge therefore have no recomputation budget to vary and yield one latency and one score on each benchmark, so each appears as a single point, not a curve. In contrast, SpecPrefill, SpecPC, and PANORAMA are evaluated at four recomputation budgets of 2.5, 5, 10, and 20%, and the four resulting points form each curve. The original legend incorrectly used line symbols, and the CacheBridge point was partly clipped by the axis. We have corrected Figure 4 to show both methods as labeled markers and made the CacheBridge point fully visible in the revised version. Please note that the revised Figure 4, now also includes the look-ahead decode and the scoring in the handoff latency at look-ahead lengths of 1 and 8 tokens, as described in our response to Reviewer v9jM.

Q3. Figure 5. Native Prefill's legend key is a dashed line, but it doesn't appear in the subfigure on the left. Why? What's the dark bars on the left of all four bars in each batch size group.

We thank the reviewer for noting the inconsistency in the legend of Figure 5. The dark bars in Figure 5(a) are Native Prefill, but the original legend incorrectly depicted Native Prefill as a dashed line. We have corrected the legend to use a matching dark bar in Figure 5 of the revised manuscript. Figure 5(b) has no Native Prefill bar because the panel reports only handoff latency, and Native Prefill is not a handoff method.

thanks for your answers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.