- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies cross-model KV cache transfer to reduce repeated prefilling when a request switches from a source model to a receiver model. CacheMorph is motivated by two observations: key and value errors affect attention differently, and query heads sharing the same KV cache respond differently to transfer errors. The method combines an asymmetric mapper, which predicts keys and conditions value prediction on the mapped keys, with a query-adaptive reader that adjusts access to transferred caches. Both components are trained using the receiver’s native-cache output distribution as supervision. A streaming pipeline overlaps cache transfer and mapping with model computation. Experiments evaluate transfer quality across model pairs and tasks, alongside latency improvements in model-switching and multi-agent workloads.
Critical strengths
-
The problem is practically relevant, and the method is well motivated. Repeated prefilling of long shared contexts can undermine the benefits of model routing. The analysis connects the different roles of K/V errors and query-head sensitivity to the proposed mapper and reader. Supervising receiver behavior also provides a reasonable objective beyond cache reconstruction alone.
-
The evaluation is broad and includes useful component ablations. The paper considers transfers across model scales and families, multiple task categories, and application workflows. Comparisons against a budget-matched MLP, alternative conditioning mechanisms, shared routing, and reader variants support the architectural choices. The multi-hop experiments and reader evaluations with other mappers also help characterize the method’s benefits and limitations.
Critical weaknesses
-
The practical quality–efficiency trade-off is not yet fully established. The paper includes receiver re-prefill and all-14B workflow comparisons, but lacks an all-A reference and a more systematic comparison of quality, latency, and cost across model-use strategies. It therefore remains unclear whether the quality improvement over continuing with the source model justifies the additional overhead, and whether the resource savings over using the receiver model throughout justify the quality loss. The variation across model pairs and degradation under multi-hop transfer further suggest that the conditions under which CacheMorph is preferable need clearer characterization.
-
Implementation details and attribution of efficiency gains remain incomplete. The procedure for selecting source layers and heads for each target KV head is insufficiently specified. The system evaluation would also benefit from clearer descriptions of its execution setting and the Text-only scheduling policy. Reported speedups combine avoided receiver prefill, streaming overlap, low-rank mapping, and other implementation optimizations, but controlled end-to-end comparisons do not sufficiently separate their contributions. The overlap statistics in Table 2 do not substitute for measurements with individual optimizations enabled and disabled.
Questions for authors
Algorithm and Applicability
- How are the mapper’s source features selected? For each target layer and KV head, which source layers and heads are available to the K/V branches, and how are their supports selected? Please clarify the selection criterion and number of selected sources, distinguish fixed supports from learned routing weights, and explain how the procedure handles models with different layer and KV-head counts.
- Could the paper provide a more complete comparison of model-use strategies? On the same tasks and role workflows, could the authors jointly compare all-A, the existing all-B baseline, Text-only, and CacheMorph in terms of quality, end-to-end latency, and cost? In particular, how much quality does CacheMorph gain over all-A, and how much resource usage does it save relative to all-B? Evaluating multiple configurations would help identify where CacheMorph lies on the quality–efficiency Pareto frontier.
- What is the cost of extending pairwise transfer to a model pool? How much training data, GPU time, and storage are required for each mapper/reader pair? Which components must be retrained when a model is added or updated? How do these costs scale with the supported routing graph?
- How should multi-hop degradation affect routing decisions? The paper observes substantial quality degradation for some cross-family transfer chains. Under what conditions should a system choose direct transfer, multi-hop transfer, or receiver re-prefill? Could these results be used to characterize the method’s practical operating range more explicitly?
System Evaluation
- What are the specific settings of the system performance evaluation? Please clarify the execution environment and workload conditions under which latency and speedups are measured, including the inference framework and basic runtime settings, to help understand the applicability of the reported results.
- How are the two models scheduled in the Text-only baseline? When the shared history is available upfront, the receiver is predetermined, and the models reside on separate GPUs, are the source and receiver allowed to prefill concurrently? If execution is sequential, what data or routing dependency requires this ordering? A concurrent-prefill baseline, where applicable, would help distinguish cache-transfer benefits from scheduling effects.
- How much does each optimization contribute to the end-to-end speedup? Could controlled experiments separate the contributions of avoiding receiver prefill, streaming/overlap, low-rank mapping, and other implementation optimizations? A comparison of the same CacheMorph implementation with overlap enabled and disabled would be particularly useful. Please also make explicit whether source prefill is included in each reported latency metric and distinguish cold requests from handoffs with an existing source cache, so that the relationship between component timings and overall speedups is clear.
4 comments
Response to Reviewer jjD4
Response to Systems Q1: Hardware, Software, and Measurement Protocol
We have fully documented the controlled system benchmark setup in the revised manuscript:
- Environment and Precision: Experiments run on two resident NVIDIA H200 NVL GPUs with PyTorch 2.13.0 and CUDA 13.0 using packed variable-length attention. Backbone models and KV caches use BF16, while mapping weights use FP32. All evaluated mapping methods share the exact same paging, token-alignment, buffer, and scheduling implementation.
- Measurement Protocol: Each context length is evaluated on 10 fixed inputs across 3 runs after 2 warmup iterations, with randomized method ordering and continuous GPU process monitoring.
- Timing Boundaries: Cold-request timing starts prior to source prefill, whereas pre-existing cache handoff timing starts after the source cache is built and synchronized. Both include online transfer, KV mapping, and target first-token generation. Controlled fixed-length latency benchmarks and open-ended task quality evaluations are reported separately.
Response to Systems Q2: Sequential vs. Parallel Text-Only Prefill
We implemented and benchmarked Parallel Text-only prefill (applicable when both full history and target model assignment are known upfront across separate GPUs) and clarified the two distinct scheduling regimes:
- Pre-existing Source Cache Handoffs: When source execution has already completed (e.g., conditional escalation or multi-turn routing), source prefill cost is sunk. Comparing cache transfer directly against target re-prefill, CacheMorph delivers 4.3x-6.8x handoff speedups.
- Upfront Dual-GPU Execution: In the full RAG workflow, CacheMorph+Reader still achieves a 1.42x end-to-end speedup over Parallel Text-only (1.089s vs. 1.541s, while cutting token cost by 54.44%). On single-turn cold-request TTFT, speedups over Parallel Text-only are 1.28x and 0.81x across the two transfer directions; the latter reflects cases where the target model is predetermined upfront with faster native prefill, where direct target execution is naturally selected.
Response to W2 and Systems Q3: Factorized Ablation of System and Architectural Components
We conducted factorized ablations to isolate prefill avoidance, pipelining, shared execution, and parameter compression:
- Prefill Avoidance vs. Pipelining: Our primary handoff speedups use dense checkpoints and do not rely on low-rank compression. Holding checkpoints, kernels, and page settings fixed while disabling pipeline overlap increases cold-request TTFT from 1670.25 to 1885.19 ms (1.13x pipeline gain) and from 2375.68 to 2729.32 ms (1.15x pipeline gain) across the two directions, cleanly separating pipeline overlap from target prefill elimination.
- Shared Source-Feature Execution: Holding a compact Mapper checkpoint (175.86M learned parameters) fixed and enabling shared source-feature reading and projection reduces Qasper TTFT from 106.00 to 88.41 ms (1.20x speedup, paired CI [1.15, 1.25]) and yields 1.04x on LCC (CI [0.99, 1.09]) across 48 fixed inputs per task.
- Dense vs. Compact Capacity Spectrum: While the dense parent Mapper (420.05M parameters; 38.27 Qasper / 63.88 LCC) maximizes quality, the compact 175.86M Mapper (26.49 Qasper / 48.06 LCC; unchanged KV transfer bytes) uses 6.94% fewer learned parameters than the fixed C2C-style checkpoint while outperforming it by +11.63 on Qasper and +29.25 on LCC at comparable TTFT (+2.66 and +4.03 ms). Parameter counts, layout speedups, and accuracy trade-offs are reported side-by-side in the revision.
Thank you for the detailed responses and additional experiments. The new quality-efficiency comparisons, system ablations, and clarifications on implementation details address most of my original questions.
However, the practical quality-efficiency gains appear relatively modest and are mainly observed in specific workloads. I therefore decide to maintain my original score.
Response to the quality-efficiency concern
Thank you for the follow-up. While we agree that end-to-end gains depend on the workload, we respectfully push back on the characterization that the practical quality-efficiency gains are modest in the regimes CacheMorph targets.
Workload dependence follows directly from the computation CacheMorph eliminates. When a source model has already prefilled a long context and hands off to a stronger responder, the target model must ordinarily re-prefill that entire shared prefix. Across 100 MuSiQue RAG requests, running the 4B model throughout achieves 30.32 F1 in 0.82 s, whereas running the 14B model throughout achieves 42.08 F1 in 1.57 s (and text-only escalation takes 2.15 s). CacheMorph reaches 36.97 F1 in 1.09 s end-to-end, capturing 57% of the available F1 gain from upgrading to 14B at only 36% of the additional latency. Conversely, in the MATH-500 workflow, autoregressive decoding accounts for 99.6% of text-only runtime, which naturally bounds the end-to-end impact of saving prefix computation.
Substantial quality-efficiency gains also extend well beyond RAG. On cross-family transfer (Qwen3-8B -> Llama-3.1-8B), we evaluated native target prefill alongside output-adapted Ridge, C2C-style mapping, and CacheMorph (with and without its Reader), with all transfer methods sharing the exact same streaming backend. Across these five configurations, CacheMorph sits on the empirical quality-latency Pareto frontier on five of six tasks, measured by both TTFT and end-to-end completion time. On Qasper, the CacheMorph Mapper alone achieves the highest mean F1 (45.99 vs. 38.54 for native target prefill) while cutting mean completion time by 39% (from 511 to 310 ms). On LCC, adding the Reader lifts CacheMorph to 63.52 vs. 43.46 for adapted Ridge (the strongest competing mapper on this task), delivering a +20.06 point gain (paired 95% CI [11.79, 28.06]) for just 19.10 ms of additional TTFT.
Together, these results establish a clear, quantitative basis for deploying cache transfer: whenever a target model would otherwise re-prefill substantial context, CacheMorph markedly advances the achievable quality-latency frontier across document QA, code completion, and end-to-end RAG workflows.
Response to Reviewer jjD4
Response to W2 and Algorithmic Q1: Source Layer/Head Selection and Fixed Support vs. Learned Routing
We have detailed the exact checkpoint selection protocol used across our main and system evaluations:
Response to W1 and Algorithmic Q2: Quality, Latency, and Cost vs. All-A, All-B, and Text-Only
We added controlled end-to-end workflow comparisons across All-A, All-B, sequential Text-only, Parallel Text-only (for workflows with upfront inputs), CacheMorph Mapper, and Mapper+Reader.
On RAG (100 fixed MuSiQue inputs, Qwen3-4B as Reader -> Qwen3-14B as Responder), quality and latency from live timed outputs are:
CacheMorph (Mapper+Reader) establishes an effective quality-latency operating point between All-A and All-B:
We evaluated a 100-problem MATH-500 Solver-Verifier workflow. Accuracies for All-A, All-B, Text-only, Mapper, and Mapper+Reader are 81%, 86%, 83%, 81%, and 80%, with latencies of 32.57, 42.59, 37.76, 37.00, and 37.32 s. Because the verifier history prefix averages only 976 tokens and autoregressive decoding accounts for 99.6% of Text-only runtime, Amdahl's Law naturally bounds prefill-transfer gains here. Together, these two workflows clearly define CacheMorph's target regime: substantial quality-latency gains on long-context, prefill-intensive RAG workflows versus decode-dominated short-prefix tasks.
Response to Algorithmic Q3: Scaling to a Model Pool
In a routing graph with |E| supported directed edges, artifacts and training effort scale linearly with |E| rather than all model pairs. Each directed edge uses one Mapper and a companion Reader:
Recorded Mapper optimization uses 16,000 and 18,048 training items totaling 19.31 and 21.75 single-GPU hours; Reader training uses 128 samples and 1,500 updates taking 0.148 and 0.131 hours (~8-9 minutes). Adding or updating a model only requires fitting its incident edges while reusing all other existing edges. Parameter storage, optimization hours, and offline preparation are reported distinctly in the revision.
Response to Algorithmic Q4: Multi-Hop Routing Decisions
Relative to direct transfer, 2-hop intra-family chains shift by 1.40 (ContextQA) and 4.61 (Reasoning) points, while cross-family 2-hop chains shift by 9.52 and 13.54 points. We formulate an offline profile-guided selection rule validated per workload: (1) use direct edges whenever available; (2) use multi-hop paths to expand reachability or reduce measured cost when the full path satisfies target quality thresholds; and (3) route to native B prefill as an independent execution strategy when strict native accuracy is required. All reported CacheMorph results reflect pure cache transfer without mixing fallback execution.