acceptodds
Under review as a conference paper at ICLR 2027

CacheMorph: Asymmetric KV Transfer for Efficient Model Switching

20 views

Abstract

Multi-model LLM systems often move a request between models, and routers switch models as load changes. Each switch has a hidden cost: the receiving model must re-prefill a shared context that may be tens of thousands of tokens long. Reusing the source model's key-value (KV) cache could remove this cost. However, same-model reuse methods do not directly support such switches, while cross-model cache alignment can still incur substantial accuracy losses. Aligning KV representations alone does not ensure accurate receiver outputs. We show why this gap matters: key errors change where attention reads, value errors change the content it retrieves, and query heads sharing KV respond differently to the same errors. Based on these insights, we introduce CacheMorph, a framework that enables the receiver to skip prefill of shared history by translating KV caches across models. Its asymmetric mapper predicts keys and values separately, conditioning each value on its mapped key. A lightweight, query-adaptive reader then adjusts how each head accesses the shared cache. Both components are trained to match the frozen receiver's native-cache output distribution. A streaming pipeline overlaps cache transfer and mapping with ongoing computation. CacheMorph improves accuracy over prior cross-model KV reuse methods, especially across model families. At 32K tokens, it achieves up to a time-to-first-token speedup over receiver re-prefill. In RAG cascades, it delivers approximately speedup and over 50% lower estimated token cost while retaining over 90% of re-prefill accuracy.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Cross-model KV Cache Reuse

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jjD4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies cross-model KV cache transfer to reduce repeated prefilling when a request switches from a source model to a receiver model. CacheMorph is motivated by two observations: key and value errors affect attention differently, and query heads sharing the same KV cache respond differently to transfer errors. The method combines an asymmetric mapper, which predicts keys and conditions value prediction on the mapped keys, with a query-adaptive reader that adjusts access to transferred caches. Both components are trained using the receiver’s native-cache output distribution as supervision. A streaming pipeline overlaps cache transfer and mapping with model computation. Experiments evaluate transfer quality across model pairs and tasks, alongside latency improvements in model-switching and multi-agent workloads.

Critical strengths
  1. The problem is practically relevant, and the method is well motivated. Repeated prefilling of long shared contexts can undermine the benefits of model routing. The analysis connects the different roles of K/V errors and query-head sensitivity to the proposed mapper and reader. Supervising receiver behavior also provides a reasonable objective beyond cache reconstruction alone.

  2. The evaluation is broad and includes useful component ablations. The paper considers transfers across model scales and families, multiple task categories, and application workflows. Comparisons against a budget-matched MLP, alternative conditioning mechanisms, shared routing, and reader variants support the architectural choices. The multi-hop experiments and reader evaluations with other mappers also help characterize the method’s benefits and limitations.

Critical weaknesses
  1. The practical quality–efficiency trade-off is not yet fully established. The paper includes receiver re-prefill and all-14B workflow comparisons, but lacks an all-A reference and a more systematic comparison of quality, latency, and cost across model-use strategies. It therefore remains unclear whether the quality improvement over continuing with the source model justifies the additional overhead, and whether the resource savings over using the receiver model throughout justify the quality loss. The variation across model pairs and degradation under multi-hop transfer further suggest that the conditions under which CacheMorph is preferable need clearer characterization.

  2. Implementation details and attribution of efficiency gains remain incomplete. The procedure for selecting source layers and heads for each target KV head is insufficiently specified. The system evaluation would also benefit from clearer descriptions of its execution setting and the Text-only scheduling policy. Reported speedups combine avoided receiver prefill, streaming overlap, low-rank mapping, and other implementation optimizations, but controlled end-to-end comparisons do not sufficiently separate their contributions. The overlap statistics in Table 2 do not substitute for measurements with individual optimizations enabled and disabled.

Questions for authors

Algorithm and Applicability

  1. How are the mapper’s source features selected? For each target layer and KV head, which source layers and heads are available to the K/V branches, and how are their supports selected? Please clarify the selection criterion and number of selected sources, distinguish fixed supports from learned routing weights, and explain how the procedure handles models with different layer and KV-head counts.
  2. Could the paper provide a more complete comparison of model-use strategies? On the same tasks and role workflows, could the authors jointly compare all-A, the existing all-B baseline, Text-only, and CacheMorph in terms of quality, end-to-end latency, and cost? In particular, how much quality does CacheMorph gain over all-A, and how much resource usage does it save relative to all-B? Evaluating multiple configurations would help identify where CacheMorph lies on the quality–efficiency Pareto frontier.
  3. What is the cost of extending pairwise transfer to a model pool? How much training data, GPU time, and storage are required for each mapper/reader pair? Which components must be retrained when a model is added or updated? How do these costs scale with the supported routing graph?
  4. How should multi-hop degradation affect routing decisions? The paper observes substantial quality degradation for some cross-family transfer chains. Under what conditions should a system choose direct transfer, multi-hop transfer, or receiver re-prefill? Could these results be used to characterize the method’s practical operating range more explicitly?

System Evaluation

  1. What are the specific settings of the system performance evaluation? Please clarify the execution environment and workload conditions under which latency and speedups are measured, including the inference framework and basic runtime settings, to help understand the applicability of the reported results.
  2. How are the two models scheduled in the Text-only baseline? When the shared history is available upfront, the receiver is predetermined, and the models reside on separate GPUs, are the source and receiver allowed to prefill concurrently? If execution is sequential, what data or routing dependency requires this ordering? A concurrent-prefill baseline, where applicable, would help distinguish cache-transfer benefits from scheduling effects.
  3. How much does each optimization contribute to the end-to-end speedup? Could controlled experiments separate the contributions of avoiding receiver prefill, streaming/overlap, low-rank mapping, and other implementation optimizations? A comparison of the same CacheMorph implementation with overlap enabled and disabled would be particularly useful. Please also make explicit whether source prefill is included in each reported latency metric and distinguish cold requests from handoffs with an existing source cache, so that the relationship between component timings and overall speedups is clear.
4 comments
AuthorsOpenReview ↗

Response to Reviewer jjD4

Response to W2 and Algorithmic Q1: Source Layer/Head Selection and Fixed Support vs. Learned Routing

We have detailed the exact checkpoint selection protocol used across our main and system evaluations:

  • Calibration-Driven Support Selection: For each KV head in each target layer, the K and V branches independently select 8 source layers, where each selected source layer provides all of its KV heads. Using strictly offline calibration data (4,096 token positions across 500 documents), we fit ridge-regression probes from each candidate source layer to the target head and rank candidates by mean explained variance across target coordinates. We then fit an affine predictor over the selected support set using 128,000 positions.
  • Fixed Support vs. Learned Routing: The support set determines which source features are accessible and remains fixed during training, whereas routing weights determine the contribution of those features and are learned during subsequent optimization.
  • Heterogeneous Architecture Support: Rectangular projections map source features to target heads, bridging models without requiring matching layer counts, KV head counts, head dimensions, or 1-to-1 head correspondence. Separate full-source-layer architectural explorations are explicitly marked and kept distinct from these main system checkpoints.

Response to W1 and Algorithmic Q2: Quality, Latency, and Cost vs. All-A, All-B, and Text-Only

We added controlled end-to-end workflow comparisons across All-A, All-B, sequential Text-only, Parallel Text-only (for workflows with upfront inputs), CacheMorph Mapper, and Mapper+Reader.

On RAG (100 fixed MuSiQue inputs, Qwen3-4B as Reader -> Qwen3-14B as Responder), quality and latency from live timed outputs are:

StrategyEMF1Latency (s)Token Cost (cents)
All-A24.0030.320.8200.1326
All-B32.0042.081.5700.2863
Text-only32.0042.082.1510.3488
Parallel Text-only32.0042.081.5410.3488
Mapper only27.0035.301.0720.1590
Mapper+Reader29.0036.971.0890.1589

CacheMorph (Mapper+Reader) establishes an effective quality-latency operating point between All-A and All-B:

  • Quality and Speedup: It improves F1 by +6.66 (+5.00 EM) over All-A (with 32.78% higher latency), while achieving a 1.44x speedup over All-B (within 3.00 EM and 5.11 F1) and a 1.42x speedup over Parallel Text-only.
  • Cost and GPU Occupancy: Under uniform cached-token read pricing, reference token cost is 44.48% lower than All-B and 54.44% lower than Text-only. Under a 1-GPU native and 2-GPU cascade reservation model, Mapper+Reader also reduces total GPU occupancy by 29.4% compared to Parallel Text-only (2.177 vs. 3.082 GPU-slot-seconds; single-GPU All-B is 1.570).

We evaluated a 100-problem MATH-500 Solver-Verifier workflow. Accuracies for All-A, All-B, Text-only, Mapper, and Mapper+Reader are 81%, 86%, 83%, 81%, and 80%, with latencies of 32.57, 42.59, 37.76, 37.00, and 37.32 s. Because the verifier history prefix averages only 976 tokens and autoregressive decoding accounts for 99.6% of Text-only runtime, Amdahl's Law naturally bounds prefill-transfer gains here. Together, these two workflows clearly define CacheMorph's target regime: substantial quality-latency gains on long-context, prefill-intensive RAG workflows versus decode-dominated short-prefix tasks.

Response to Algorithmic Q3: Scaling to a Model Pool

In a routing graph with |E| supported directed edges, artifacts and training effort scale linearly with |E| rather than all model pairs. Each directed edge uses one Mapper and a companion Reader:

ComponentQwen2.5 -> Qwen3Qwen3 -> Llama
Dense Mapper (Params / FP32)420.05M / 1602.35 MiB742.49M / 2832.38 MiB
Reader (Params / FP32)0.590M / 2.25 MiB0.524M / 2.00 MiB

Recorded Mapper optimization uses 16,000 and 18,048 training items totaling 19.31 and 21.75 single-GPU hours; Reader training uses 128 samples and 1,500 updates taking 0.148 and 0.131 hours (~8-9 minutes). Adding or updating a model only requires fitting its incident edges while reusing all other existing edges. Parameter storage, optimization hours, and offline preparation are reported distinctly in the revision.

Response to Algorithmic Q4: Multi-Hop Routing Decisions

Relative to direct transfer, 2-hop intra-family chains shift by 1.40 (ContextQA) and 4.61 (Reasoning) points, while cross-family 2-hop chains shift by 9.52 and 13.54 points. We formulate an offline profile-guided selection rule validated per workload: (1) use direct edges whenever available; (2) use multi-hop paths to expand reachability or reduce measured cost when the full path satisfies target quality thresholds; and (3) route to native B prefill as an independent execution strategy when strict native accuracy is required. All reported CacheMorph results reflect pure cache transfer without mixing fallback execution.

AuthorsOpenReview ↗

Response to Reviewer jjD4

Response to Systems Q1: Hardware, Software, and Measurement Protocol

We have fully documented the controlled system benchmark setup in the revised manuscript:

  • Environment and Precision: Experiments run on two resident NVIDIA H200 NVL GPUs with PyTorch 2.13.0 and CUDA 13.0 using packed variable-length attention. Backbone models and KV caches use BF16, while mapping weights use FP32. All evaluated mapping methods share the exact same paging, token-alignment, buffer, and scheduling implementation.
  • Measurement Protocol: Each context length is evaluated on 10 fixed inputs across 3 runs after 2 warmup iterations, with randomized method ordering and continuous GPU process monitoring.
  • Timing Boundaries: Cold-request timing starts prior to source prefill, whereas pre-existing cache handoff timing starts after the source cache is built and synchronized. Both include online transfer, KV mapping, and target first-token generation. Controlled fixed-length latency benchmarks and open-ended task quality evaluations are reported separately.

Response to Systems Q2: Sequential vs. Parallel Text-Only Prefill

We implemented and benchmarked Parallel Text-only prefill (applicable when both full history and target model assignment are known upfront across separate GPUs) and clarified the two distinct scheduling regimes:

  • Pre-existing Source Cache Handoffs: When source execution has already completed (e.g., conditional escalation or multi-turn routing), source prefill cost is sunk. Comparing cache transfer directly against target re-prefill, CacheMorph delivers 4.3x-6.8x handoff speedups.
  • Upfront Dual-GPU Execution: In the full RAG workflow, CacheMorph+Reader still achieves a 1.42x end-to-end speedup over Parallel Text-only (1.089s vs. 1.541s, while cutting token cost by 54.44%). On single-turn cold-request TTFT, speedups over Parallel Text-only are 1.28x and 0.81x across the two transfer directions; the latter reflects cases where the target model is predetermined upfront with faster native prefill, where direct target execution is naturally selected.

Response to W2 and Systems Q3: Factorized Ablation of System and Architectural Components

We conducted factorized ablations to isolate prefill avoidance, pipelining, shared execution, and parameter compression:

  • Prefill Avoidance vs. Pipelining: Our primary handoff speedups use dense checkpoints and do not rely on low-rank compression. Holding checkpoints, kernels, and page settings fixed while disabling pipeline overlap increases cold-request TTFT from 1670.25 to 1885.19 ms (1.13x pipeline gain) and from 2375.68 to 2729.32 ms (1.15x pipeline gain) across the two directions, cleanly separating pipeline overlap from target prefill elimination.
  • Shared Source-Feature Execution: Holding a compact Mapper checkpoint (175.86M learned parameters) fixed and enabling shared source-feature reading and projection reduces Qasper TTFT from 106.00 to 88.41 ms (1.20x speedup, paired CI [1.15, 1.25]) and yields 1.04x on LCC (CI [0.99, 1.09]) across 48 fixed inputs per task.
  • Dense vs. Compact Capacity Spectrum: While the dense parent Mapper (420.05M parameters; 38.27 Qasper / 63.88 LCC) maximizes quality, the compact 175.86M Mapper (26.49 Qasper / 48.06 LCC; unchanged KV transfer bytes) uses 6.94% fewer learned parameters than the fixed C2C-style checkpoint while outperforming it by +11.63 on Qasper and +29.25 on LCC at comparable TTFT (+2.66 and +4.03 ms). Parameter counts, layout speedups, and accuracy trade-offs are reported side-by-side in the revision.

Thank you for the detailed responses and additional experiments. The new quality-efficiency comparisons, system ablations, and clarifications on implementation details address most of my original questions.

However, the practical quality-efficiency gains appear relatively modest and are mainly observed in specific workloads. I therefore decide to maintain my original score.

AuthorsOpenReview ↗

Response to the quality-efficiency concern

Thank you for the follow-up. While we agree that end-to-end gains depend on the workload, we respectfully push back on the characterization that the practical quality-efficiency gains are modest in the regimes CacheMorph targets.

Workload dependence follows directly from the computation CacheMorph eliminates. When a source model has already prefilled a long context and hands off to a stronger responder, the target model must ordinarily re-prefill that entire shared prefix. Across 100 MuSiQue RAG requests, running the 4B model throughout achieves 30.32 F1 in 0.82 s, whereas running the 14B model throughout achieves 42.08 F1 in 1.57 s (and text-only escalation takes 2.15 s). CacheMorph reaches 36.97 F1 in 1.09 s end-to-end, capturing 57% of the available F1 gain from upgrading to 14B at only 36% of the additional latency. Conversely, in the MATH-500 workflow, autoregressive decoding accounts for 99.6% of text-only runtime, which naturally bounds the end-to-end impact of saving prefix computation.

Substantial quality-efficiency gains also extend well beyond RAG. On cross-family transfer (Qwen3-8B -> Llama-3.1-8B), we evaluated native target prefill alongside output-adapted Ridge, C2C-style mapping, and CacheMorph (with and without its Reader), with all transfer methods sharing the exact same streaming backend. Across these five configurations, CacheMorph sits on the empirical quality-latency Pareto frontier on five of six tasks, measured by both TTFT and end-to-end completion time. On Qasper, the CacheMorph Mapper alone achieves the highest mean F1 (45.99 vs. 38.54 for native target prefill) while cutting mean completion time by 39% (from 511 to 310 ms). On LCC, adding the Reader lifts CacheMorph to 63.52 vs. 43.46 for adapted Ridge (the strongest competing mapper on this task), delivering a +20.06 point gain (paired 95% CI [11.79, 28.06]) for just 19.10 ms of additional TTFT.

Together, these results establish a clear, quantitative basis for deploying cache transfer: whenever a target model would otherwise re-prefill substantial context, CacheMorph markedly advances the achievable quality-latency frontier across document QA, code completion, and end-to-end RAG workflows.

Reviewer_aQ1nOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CacheMorph, a framework for transferring KV caches across different LLMs to avoid receiver-side re-prefilling during model switching. The method combines an asymmetric mapper that predicts keys and values separately, with value prediction conditioned on mapped keys, and a query-adaptive reader that applies low-rank group- and head-specific query corrections. A streaming backend overlaps cache transfer, mapping, and model computation. Experiments across six model pairs and 14 tasks show consistent improvements over the evaluated cache-transfer baselines. At a 32K-token prefix, CacheMorph reports up to a 4.12× TTFT speedup over receiver re-prefill.

Critical strengths
  1. The paper addresses an important practical bottleneck in routed and multi-model LLM systems. The distinction between key addressing errors and value content errors provides a clear motivation for asymmetric K/V mapping.

  2. Training against the frozen receiver’s output distribution is better aligned with downstream behavior than minimizing KV reconstruction error alone. The query-adaptive reader is a useful design that handles head-specific sensitivity without duplicating physical KV caches.

  3. The ablations generally support the main components, including separate K/V routes, key-conditioned value prediction, functional supervision, and head-specific query correction. The evaluation covers multiple model families, task categories, and end-to-end workflows.

Critical weaknesses
  1. The latency comparison may confound the method with its execution backend. CacheMorph is the only method in Table 2 that benefits from substantial streaming overlap. C2C-style and Ridge also appear positionwise and query-independent, and may therefore admit similar streaming execution. Applying the same backend to all compatible methods is necessary to establish whether CacheMorph itself has the lowest TTFT.

  2. Transfer quality remains model-dependent. The average gap from native receiver performance reaches 17.9–26.9% for several transfers, and cross-family multi-hop transfer degrades substantially. The method therefore provides an accuracy-efficiency trade-off rather than generally lossless cache reuse.

  3. The systems evaluation uses two H200 NVL GPUs connected by NVLink. Performance under PCIe-only, cross-node, dynamically batched, or concurrent serving conditions remains unclear.

Questions for authors
  1. Can the same page-streaming backend be applied to Ridge and C2C-style? If so, how do the TTFT results change?

  2. Why is the reader enabled for only the initial decoding window? Please provide an ablation over the window length.

  3. How sensitive is performance to the interconnect, especially PCIe-only or cross-node settings?

2 comments
AuthorsOpenReview ↗

Response to Reviewer aQ1n

Response to W1 & Q1: Unified Backend Comparison and CacheMorph's Algorithmic Advantage

Yes—we integrated both Ridge and C2C-style baselines into the exact same paged pipeline and evaluated quality and latency head-to-head. To ensure a rigorous comparison, we strengthened both baselines using output-adapted checkpoints trained with the identical adaptation data, 256 updates, output supervision, and checkpoint selection rule as CacheMorph. Across 6 tasks (48 fixed inputs per task, 3 runs per configuration, totaling 4,320 requests), the results on Qwen3-8B -> Llama-3.1-8B-Instruct are:

TaskNative BAdapted RidgeAdapted C2C-styleCacheMorph MapperMapper+Reader
Qasper38.5438.5032.5045.9945.11
LCC67.0043.4626.4257.7363.52
HotpotQA50.1252.5248.4649.0051.43
MuSiQue27.7627.8519.1128.4032.71
GovReport34.2431.4929.4733.3634.02
Passage Retrieval100.0054.176.2572.9275.00

CacheMorph (Mapper+Reader) outperforms both strengthened baselines on 5 out of 6 tasks, preventing severe quality collapse on demanding tasks like LCC and Passage Retrieval while incurring only negligible millisecond-level mapping overhead:

  • LCC: +20.06 points over Adapted Ridge (95% CI: [11.79, 28.06]) with just 19.10 ms additional TTFT, and +37.10 points over Adapted C2C-style.
  • Passage Retrieval: Achieves 75.00, substantially outperforming Adapted Ridge (54.17) and Adapted C2C-style (6.25).
  • GovReport: +2.53 and +4.55 points over Adapted Ridge and C2C-style (95% CIs: [1.29, 3.80] and [3.15, 5.97]) for only 57.56 ms and 53.13 ms extra TTFT, respectively.
  • Qasper: Mapper alone gains +13.49 F1 over Adapted C2C-style (95% CI: [3.14, 23.92]) with only 12.63 ms extra TTFT.

These controlled comparisons demonstrate that CacheMorph delivers substantial, statistically significant quality gains on top of a shared high-efficiency execution backend, cleanly separating our algorithmic contributions from pipeline-level benefits in the revised manuscript.


Response to W2: Principled Positioning and Holistic Routing for Direct & Multi-hop Transfers

In the revised manuscript, we show that CacheMorph retains up to 99.4% of native target task-averaged scores across transfer directions (and exceeds Native B on several reasoning benchmarks). Combined with our joint quality-latency evaluation, we position CacheMorph as a profile-guided Pareto optimization framework with clear deployment criteria: practitioners validate target quality thresholds on a lightweight calibration split and select the optimal execution path among source continuation, direct transfer, multi-hop transfer, and native target prefill.

For multi-hop scenarios, our routing criterion evaluates the complete end-to-end path holistically rather than extrapolating from individual single-hop edges, ensuring multi-hop handoffs are selected strictly when end-to-end path quality and latency outperform direct or native alternatives. These selection rules and empirical comparisons are detailed in the revised manuscript.


Response to Q2: Rationale and Efficiency of the Initial Decoding Window for Reader

Reader is specifically designed to bridge the representation gap during the target model's initial reading of imported KV history immediately after handoff. As decoding progresses, newly generated native target KV states naturally anchor subsequent generation, allowing us to turn off Reader and avoid unnecessary per-step overhead.

Holding the trained Mapper and Reader frozen and varying only the active decoding window on LCC yields:

ConfigurationScoreFull Generation Latency (ms)
Mapper Only57.731229.65
Reader (First 4 steps, default)63.521257.12
Reader (First 16 steps)65.131317.48
Reader (All steps)65.131577.24

Enabling Reader for just the first 4 steps boosts quality by +5.79 points while adding only 2.2% to end-to-end latency. Extending the window to 16 steps captures 100% of the quality of all-step execution (65.13 vs. 65.13) while cutting full generation latency by 16.47%. Similarly, on GovReport, the 4-step window retains 98.8% of the all-step score (34.02 vs. 34.43) while reducing generation time from 11.17 s to 9.76 s. Thus, a short initial window captures nearly the full quality benefit at minimal cost, while offering a flexible tuning knob across workloads.

AuthorsOpenReview ↗

Response to Reviewer aQ1n

Response to W3 & Q3: Robustness Across PCIe and Host-Staged Interconnects (No NVLink Dependency)

CacheMorph does not rely on NVLink and preserves its high speedups over standard PCIe and even CPU-memory-staged transfers. We conducted 2,400 requests across 8K and 32K context lengths over a pure PCIe interconnect (GPUs connected via a PCIe host bridge within the same NUMA node, with zero NVLink connectivity), using identical checkpoints, inputs, and execution code, producing identical per-request outputs. Under 32K pre-existing source caches:

Transfer DirectionCacheMorph+Reader TTFTNative B TTFTTTFT Speedup
Qwen2.5-7B -> Qwen3-8B315.13 ms2124.36 ms6.74x
Qwen3-8B -> Llama-3.1-8B438.86 ms1901.80 ms4.33x

The paired 95% CIs for speedup are [6.72, 6.76] and [4.32, 4.34]. Compared to NVLink, CacheMorph+Reader TTFT changes by only 1.33% and 4.03%, respectively. Furthermore, in an additional stress test routing transfers through pinned CPU memory (incorporating two full DMA copies and buffer preparation in the timing), CacheMorph+Reader still achieves 6.62x and 4.15x handoff speedups across the two directions, confirming strong robustness on standard PCIe hardware.

Reviewer_gHmHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

CacheMorph addresses the repeated prefill cost incurred when an LLM request moves between models. It translates the source model’s KV cache into a cache usable by the receiver, allowing the receiver to continue without reprocessing the shared history. The mapper predicts keys and values separately, with value prediction conditioned on the mapped keys. A lightweight reader then adjusts individual query heads to improve access to the transferred cache. Both backbone models remain frozen, while the mapper and reader are trained to match the receiver’s output distributions under its native cache. The system also overlaps cache transfer and mapping with model computation. Evaluation covers six directed model transfers and 14 tasks, alongside RAG and reasoning workflows. Results show improved accuracy over the evaluated transfer baselines and up to 4.1× TTFT speedup at 32K tokens, although the remaining quality loss varies substantially across model pairs.

Critical strengths
  1. Repeated prefill is a meaningful bottleneck in multi-model workflows, and the reported gains on long inputs show that cache transfer is worth investigating.

  2. The distinction between addressing and content errors provides a sensible design rationale. The matched MLP and reader ablations offer useful evidence that the proposed components contribute beyond simply adding mapper capacity.

Critical weaknesses
  1. Transfer quality remains uneven: the average score loss relative to receiver re-prefill ranges from 0.6% to 26.9%, and errors accumulate across transfers. This matters because switching to a stronger model often aims to improve reliability. The paper does not establish when the savings justify the quality loss or how unreliable transfers could be detected.

  2. The practical efficiency advantage is not sufficiently established. Table 2 reports overlap only for CacheMorph, making the gains over other transfer methods difficult to separate from scheduling improvements. Results with equally pipelined baselines, existing source caches, and concurrent requests would better justify the additional training and serving complexity.

Questions for authors

Please refer to Weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer gHmH

Response to W1: Evaluating Quality-Overhead Trade-offs and Identifying Unreliable Transfers

Thank you for highlighting the importance of linking transfer quality to practical inference gains. We added a joint evaluation of generation quality, TTFT, and end-to-end latency on identical inputs, demonstrating that CacheMorph+Reader achieves substantial speedups while preserving—and often exceeding—native target model quality.

1. Superior Quality-Latency Trade-offs

For Qwen3-8B -> Llama-3.1-8B-Instruct (48 fixed inputs per task, averaged over 3 runs), the joint evaluation results are as follows:

TaskNative B ScoreCacheMorph+Reader ScoreNative B TTFT (ms)CacheMorph+Reader TTFT (ms)TTFT Speedup
Qasper (F1)38.5445.11165.7494.821.75x
HotpotQA (F1)50.1251.43530.44184.902.87x
MuSiQue (F1)27.7632.71694.30225.603.08x
GovReport (ROUGE-L)34.2434.02393.17144.352.72x
LCC (Edit Sim.)67.0063.52106.4386.061.24x

CacheMorph+Reader surpasses Native B in quality on Qasper, HotpotQA, and MuSiQue while delivering up to a 3.08x TTFT speedup. On GovReport, it retains 99.35% of Native B's score with a 2.72x TTFT speedup and a 1.08x end-to-end generation speedup (paired 95% confidence upper bounds on relative quality loss for Qasper and GovReport are just 0.98% and 4.56%, respectively, confirming high stability).

Furthermore, in a 100-input end-to-end RAG workflow, CacheMorph+Reader improves F1 by 6.66 points over an all-A baseline (36.97 vs. 30.32, paired 95% CI: [0.77, 12.72]), while cutting end-to-end latency by 1.44x compared to an all-B baseline (1.089s vs. 1.570s). In practice, validating on a small target calibration set allows practitioners to directly lock in these high-return configurations.

2. Lightweight Identification of Unreliable Transfers via Answer Entropy

To detect potential transfer risks without extra overhead, we introduced an answer-entropy-based risk flagging mechanism evaluated on disjoint calibration and confirmation splits. This zero-overhead signal effectively distinguishes transfer reliability with an AUROC of 0.725, boosting accuracy on accepted samples from 71.88% to 82.61% and confirming the viability of online confidence filtering.


Response to W2: Isolating Efficiency Gains—Unified Pipeline, Pre-existing Cache, and Concurrent Requests

Thank you for this constructive suggestion. We added all three controlled evaluations to strictly verify that our efficiency gains stem from the core cache transfer mechanism rather than scheduling or pipeline advantages:

  1. Unified Backend and Pipeline: We implemented Ridge, C2C-style, and CacheMorph within the exact same paging, buffering, token alignment, target cache writing, and scheduling framework, cleanly isolating the intrinsic efficiency of our mapping method from system-level overhead.

  2. Pre-existing Source Cache Handoff (32K Context): With pre-existing 32K source caches, CacheMorph+Reader achieves massive TTFT speedups over Native B re-prefill: 6.87x on Qwen2.5-7B-Instruct -> Qwen3-8B (310.99 ms vs. 2135.91 ms) and 4.54x on Qwen3-8B -> Llama-3.1-8B-Instruct (421.84 ms vs. 1913.75 ms).

  3. Concurrent Request Throughput: Under concurrent workloads with fixed A/B GPU placement (generating 16 tokens per request on dedicated receiver GPUs), system throughput increases from 0.32 and 0.36 req/s (Native B) to 0.87 and 0.86 req/s across the two transfer directions—representing a 2.72x and 2.39x throughput gain.

We have updated the main efficiency figures in the revised manuscript to reflect this unified backend comparison and included the complete experimental setup in the Appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.