acceptodds
Under review as a conference paper at ICLR 2027

SCALE-KV: State-Augmented Cross-Scale KV Cache Reuse for Long-Context Inference

Abstract

Long-context inference and context handoff in multi-agent collaboration incur substantial context-processing costs. Cross-scale key-value (KV) cache reuse allows a large model to reuse context computation performed by a smaller model, reducing repeated large-model prefilling and redundant computation after model handoff. However, structural and representational differences across model scales can degrade generation quality, while additional computation for quality recovery may offset the resulting speedup. We propose SCALE-KV, a state-augmented framework for cross-scale KV cache reuse. SCALE-KV establishes cross-model correspondence through a fixed base mapping and learns compact state-augmented residuals from the source KV cache and intermediate states. The cache converter and low-rank target-reader adaptation are jointly optimized for generation while both backbones remain frozen. At inference time, the source model prefills the full context once, after which the target directly processes questions and generates answers from the converted prefix without repeating full-context prefilling. Across three question-answering evaluations with 8–16K contexts, SCALE-KV remains within 3.53 F1 points of native large-model inference. Including source-model prefilling, SCALE-KV achieves up to time-to-first-token (TTFT) speedup for fresh requests; when the source cache and required intermediate states are available, it achieves up to TTFT speedup for handoffs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YEnAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SCALE-KV, which lets a large target model answer from a KV cache converted from a small source model, so the target does not prefill the document itself. Each target layer reads a source layer chosen by relative depth. Keys are mapped by a frozen affine map with a low rank residual after removing RoPE.Values are mapped by a fixed QR basis and a trainable affine readout. A low rank residual that also takes the source hidden states as input is added to this mapping. The converter and a LoRA on the target attention are trained jointly with answer cross entropy and distillation from native 14B, while both backbones stay frozen.

Critical strengths
  • The cost accounting is end to end. Fresh request TTFT includes source prefill, conversion, and target suffix processing. Handoffs are reported separately, and a latency matched mixture of native 4B and 14B is given as a reference.

  • The results show that reading a converted cache benefits from adapting the target reader. Retraining without reader LoRA lowers accuracy, and mapping baselines without target side adaptation fall below native 4B.

Critical weaknesses
  • The proposed converter design is not supported by the paper's own controlled comparison. Adding source hidden states, the component the method is named after, gives no measurable gain at matched capacity. In the same controlled setting, a plain linear KV map with the same reader LoRA and losses (A2) performs at least as well. The advantage over A2 in Table 1 compares the main checkpoint, whose recipe is compared with alternatives only on the test panels, against A2 from the controlled runs. The parameter reduction applies only to trainable parameters. The evidence therefore supports linear mapping plus reader LoRA, and shows no advantage for the state augmented design.

  • The improvement over simple alternatives is limited or confounded. The native 4B reference receives no task adaptation, while SCALE-KV is trained with gold answers, and removing this supervision brings it close to native 4B on HotpotQA. SCALE-KV falls below native 4B on LV-Eval, and its gain over a latency matched random mix of 4B and 14B is inconsistent across panels. The multi question comparison also omits prefix caching for native 14B, and all latency measurements use batch size 1 without a serving framework.

Questions for authors
  • What data are the converter and reader trained on, and how was the main recipe selected?

  • Native 4B fine tuned on the same data and losses, evaluated on all panels.

  • A2 trained under the main recipe and normalization, compared with SCALE-KV across multiple seeds.

  • Latency comparison against native 14B with prefix caching in a serving framework such as vLLM or SGLang.

7 comments

Thank you for the precise review and the concrete requests, and for noting the end-to-end cost accounting and the role of reader adaptation. We ran each requested experiment:

  • A2 trained under the main recipe and normalization, compared with SCALE-KV over three seeds on both sides (Q3);
  • the 4B fine-tuned on the same data and losses, evaluated on all seven panels (Q2);
  • vLLM serving against native 14B with automatic prefix caching, plus batched throughput (Q4);
  • the training-data composition and its screening against the test panels (Q1).

We also added a matched-capacity full-width hidden-state experiment, first-token latency for A2, a document-mismatch control and a recomputed latency-matched mixture (W1, W2). The table after the next two paragraphs maps each of your points to our answer.

Where SCALE-KV is used. SCALE-KV lets a large model take over a small model's reading of a long document, removing the large model's own document prefill. This handoff already occurs in deployed systems, where a lighter model processes the content before a stronger one answers:

  • Heterogeneous multi-agent systems. Lighter subagents search and read, and a stronger lead agent answers (e.g., Anthropic's multi-agent research system, with Claude Sonnet 4 subagents and a Claude Opus 4 lead; Chain of Agents, NeurIPS 2024). Today these systems hand over text; a 1,024-token text handoff falls far below SCALE-KV on every panel of Table 1.
  • Fast/slow routing and escalation. A fast model has already read the conversation or document when a stronger model takes over (the GPT-5 router in ChatGPT; RouteLLM, ICLR 2025; FrugalGPT cascades).
  • Documents processed before questions arrive. Precomputing document caches ahead of queries is established practice (TurboRAG, EMNLP 2025; CacheBlend, EuroSys 2025). SCALE-KV lets the small model do this step and hands the result to the large model when a question comes.

In all three, the large model has not read the document, so its own prefix cache cannot help. SCALE-KV removes exactly this prefill: 1.4–3.2 s per document in our vLLM measurements, with the first token up to 6.54× faster than native 14B with prefix caching.

A better fit of the frozen base map; the method is unchanged. We refined how SCALE-KV's frozen base key map is fitted (W1 / Q3 below). Nothing else in the method changes: the same components, the same 40.7M trainable parameters, training data, recipe and step-900 checkpoint, and latency within 1%. The revised Table 1 will report SCALE-KV with the refined fit, next to the original fit. SCALE-KV quality numbers below use the refined fit and, unless marked otherwise, the pre-specified training seed (20260920); the A2 reference is averaged over three training seeds.

Summary of our responses

Your concernOur answerKey evidence
W1 / Q3: A2 under the main recipe, across seedsSCALE-KV matches A2 with the original fit and is significantly better with the refined fit, for every seed.+0.04 F1 (three seeds each); +3.19 / +1.63 / +2.41 F1 per seed.
W1: Only trainable parameters are reducedThe saving is in training memory; first-token latency is on the same level as A2.81M versus 215M optimizer-state values.
W1: Hidden-state inputs at matched capacityH_s is the more informative input; a full-width path turns this into a task gain in the pre-specified run.Probes and closed-form fits; +1.96 F1 over a capacity-matched control.
W1 / Q1: Recipe and checkpoint selectionSCALE-KV and A2 share one recipe and checkpoint rule; the base-map coordinate was chosen on a disjoint tuning panel.Step 900 for all models.
Q2 / W2: Same-data adapted 4B, LV-EvalOn par with the adapted 4B while keeping the 14B; LV-Eval gap to native 4B closed.−0.58 F1; +0.01 F1 on LV-Eval; wrong-document control.
W2: Latency-matched random mixtureSCALE-KV exceeds the mixture on every timing subset.+6.41 / +6.66 / +7.95 F1 (submission: +1.41 / +5.73 / +0.08).
Q4 / W2: Prefix caching, batching, serving frameworkThe saving holds in vLLM with prefix caching enabled for native 14B.Up to 6.54× (handoff) / 1.53× (fresh) faster first token; batching barely amortizes prefill.
Q1: Training data1,941 contexts, screened against all test panels before training.50-word-window and title screen with planted positive controls.

W1 / Q3 — A2 under the main recipe, across seeds

Answer. Under the main recipe with three seeds on both sides, SCALE-KV matches A2 with the original fit and, with the refined fit, is significantly better than A2 for every seed.

As requested, we retrained A2 with SCALE-KV's main recipe and normalization and evaluated all seven panels, using three training seeds on both sides. With the base map as originally fitted, SCALE-KV already matches A2 (+0.04 F1 on the seven-panel mean, no panel significantly different) with 40.7M trainable parameters instead of 104.9M.

Fitting the frozen base key map before the target key normalization adds no trainable parameters and puts SCALE-KV +3.19 F1 above the same three-seed A2 reference, significantly better on HotpotQA 8–16K, LongBench 8–16K, HotpotQA 16–32K and LV-Eval and significantly worse on none. Every training seed reproduces this:

SCALE-KV training seedSeven-panel mean vs. A2Panels significantly better than A2Panels significantly worse than A2Seven-panel mean vs. original fit, same seed
20260920 (pre-specified)+3.19HP 8–16K, LB 8–16K, HP 16–32K, LV-Evalnone+2.91
20261107+1.63the same fournone+1.83
20261108+2.41the same fournone+2.39

HP = HotpotQA, LB = LongBench; A2 is the three-seed reference. Every difference in the two mean columns is statistically significant.

W1 — Trainable parameters and deployment cost

Answer. The saving is in training memory (81M versus 215M optimizer-state values), and first-token latency is on the same level as A2.

Trainable parametersOptimizer-state valuesFirst-token speedup over native 14B: fresh / handoffSeven-panel mean vs. A2 (F1)
A2, main recipe104.9M215M1.57× / 4.55–5.45× (deltas applied unfolded)—
SCALE-KV, refined fit40.7M81M1.64–1.65× / 5.36–6.25×+3.19

Both are timed with the same harness on the paper's timing subsets. The refined map adds no parameters and leaves latency unchanged (within 1%): the quality gain comes from a better conversion interface, not a larger training budget.

W1 — Hidden-state inputs at matched capacity

Answer. The hidden state is the more informative input, and a full-width path turns this into a +1.96 F1 gain over a capacity-matched control in the pre-specified run.

H_s is the better predictor of the target cache in every comparison we ran:

Representation-level testResult
Linear probe of the 14B's keys (held-out R²)H_s 0.733 vs. source KV 0.721
Linear probe of the 14B's values (held-out R²)H_s 0.587 vs. source KV 0.564
Closed-form map at exactly matched capacityH_s better on all 40 target layers and every tested rank

The probe advantage holds on every probed layer.

These results motivate giving the hidden state a direct, full-width path into the target representation. SCALE-KV routes H_s through a rank-64 residual on the value path; here we map the source hidden state to the target's attention input and use the target's frozen k_proj, k_norm and v_proj to construct the cache. The control replaces the hidden state with a fixed orthonormal embedding of [K_s; V_s] of the same width. The modules, parameter counts and training recipe are identical. With the pre-specified training seed 20260920, the hidden-state route improves the seven-panel mean by +1.96 F1, with a positive difference on all seven panels. Across three training seeds, the mean gain is +0.80 F1, positive on six of seven panels and significant on FRAMES 16–32K (the other two seeds give +0.55 and −0.12).

W1 / Q1 — How the recipe and checkpoint were selected

Answer. SCALE-KV and A2 now share the same recipe, normalization and checkpoint rule, and the coordinate of the refined base-map fit was chosen on a tuning panel disjoint from the test panels.

All models, including the main-recipe A2, use the fixed final checkpoint at step 900; no intermediate checkpoint is selected.

W2 / Q2 — Same-data adapted 4B, supervision and LV-Eval

Answer. SCALE-KV is on par with the same-data adapted 4B while keeping the 14B as the answering model, and its LV-Eval gap to native 4B is closed.

The content-mismatch control tests whether the 14B actually uses the transferred document. Replacing the correct document with a length-matched wrong one gives:

PanelSCALE-KV: correct documentSCALE-KV: wrong documentNative 14B: same wrong document
FRAMES 8–16K36.811.89.4
HotpotQA 8–16K64.118.520.4
LongBench47.114.512.8

With the wrong document, SCALE-KV falls to the level of native 14B reading that same wrong document (no significant difference on any panel). With the correct one, it recovers up to 101% of native 14B's document gain on the six recovery panels.

We fine-tuned the 4B with the same data, losses and LoRA setup and evaluated all seven panels. A fine-tuned 4B replaces the 14B; SCALE-KV keeps the 14B as the answering model, at comparable quality:

ReferenceSCALE-KV − reference (F1)
Same-data adapted 4B, seven-panel mean−0.58 (on par)
Native 8B, seven-panel mean+2.91
Native 4B, seven-panel mean+6.06
Native 4B, LV-Eval (the review noted a deficit of about four points)+0.01
Reviewer_QxagOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SCALE-KV, a "state"-augmented framework for cross-scale KV cache reuse. The main idea is to have a converter to construct a generation-ready context cache for the large model based on the small model's cache and intermediate state. Further optimizations are performed to reduce the conversion cost and to improve generation quality. Experimental results show that SCALE-KV's answer quality is close to Native 14B on most benchmarks, and its latency is much lower.

Critical strengths

S1. The proposed approach addresses the problem of cross-scale KV cache reuse, which is important for the performance in practice.

Critical weaknesses

W1. The paper is about "state"-augmented cross-scale KV-cache reuse, but it is not clear what states the paper refers to. Figure 1 and 2 have labels "source states". Formula (4) contains a hidden state. I wonder if these are the states the paper refers to.

W2. Table 1 shows that Native 14B has the best answer quality on six of the seven workloads, and is the second (F1 value of 40.53) on LB 16-32K. In comparison, the proposed SCALE-KV is only the best (40.65) on LB 16-32K and is dominated by Native 14B, e.g., the last column LV 16-32K, SCALE-KV is 34.57 versus Native's 14B's 42.02,.

Questions for authors

Q1. What exactly are the states the paper refers to?

4 comments

Thank you for the review and for highlighting the practical importance of cross-scale KV cache reuse. We answer both points below: we give a precise definition of the states, and we explain why native 14B is the quality ceiling SCALE-KV approaches by design, now within a small margin.

Where SCALE-KV is used. SCALE-KV lets a large model take over a small model's reading of a long document, removing the large model's own document prefill. This handoff already occurs in deployed systems, where a lighter model processes the content before a stronger one answers:

  • Heterogeneous multi-agent systems. Lighter subagents search and read, and a stronger lead agent answers (e.g., Anthropic's multi-agent research system, with Claude Sonnet 4 subagents and a Claude Opus 4 lead; Chain of Agents, NeurIPS 2024). Today these systems hand over text; a 1,024-token text handoff falls far below SCALE-KV on every panel of Table 1.
  • Fast/slow routing and escalation. A fast model has already read the conversation or document when a stronger model takes over (the GPT-5 router in ChatGPT; RouteLLM, ICLR 2025; FrugalGPT cascades).
  • Documents processed before questions arrive. Precomputing document caches ahead of queries is established practice (TurboRAG, EMNLP 2025; CacheBlend, EuroSys 2025). SCALE-KV lets the small model do this step and hands the result to the large model when a question comes.

In all three, the large model has not read the document, so its own prefix cache cannot help. SCALE-KV removes exactly this prefill: 1.4–3.2 s per document in our vLLM measurements, with the first token up to 6.54× faster than native 14B with prefix caching.

A better fit of the frozen base map; the method is unchanged. We refined how SCALE-KV's frozen base key map is fitted. Nothing else in the method changes: the same components, the same 40.7M trainable parameters, training data, recipe and step-900 checkpoint, and latency within 1%. The revised Table 1 will report SCALE-KV with the refined fit, next to the original fit. SCALE-KV quality numbers below use the refined fit and, unless marked otherwise, the pre-specified training seed (20260920).

Summary of our responses

Your concernOur answerKey evidence
W1 / Q1: What are the states?(K_s, V_s, H_s) at each aligned layer; H_s is the 4B's attention input, the hidden state in Eq. (4).H_s predicts the 14B's keys and values better on every probed layer.
W2: Quality relative to native 14BNative 14B reading the document is the quality ceiling by design; SCALE-KV stays close to it while removing that prefill.No panel significantly below native 14B; LV-Eval gap 7.45 → 3.44 F1; up to 6.54× faster first token; +2.91 F1 over native 8B at similar latency.

W1 / Q1 — What the states are

You asked what exactly the “states” in the title, Figures 1–2 and Eq. (4) refer to.

Answer. The source states are (K_s, V_s, H_s) at each aligned layer, where H_s is the 4B's attention input, i.e., the hidden state in Eq. (4).

For each aligned layer of the 4B, in the paper's notation:

SymbolDefinition
H_sthe 4B's attention input; this is the hidden state in Eq. (4)
V_sW_v H_s, the 4B's value projection
K_sk_norm(W_k H_s), the 4B's key projection
“Source states” (Figures 1–2)(K_s, V_s, H_s)

The revision will use this definition consistently across the figures, equations and implementation description.

The hidden state is the informative part of these states: held-out linear probes predict the 14B's keys and values more accurately from H_s than from the source KV representation on every probed layer, and at exactly matched capacity the closed-form hidden-state map fits the target cache better on all 40 target layers and every tested rank.

W2 — Quality relative to native 14B

You noted that native 14B is best on six of seven panels, with the largest gap on LV-Eval (34.57 versus 42.02).

Answer. Native 14B, which reads the whole document itself, is the quality ceiling SCALE-KV approaches by design: SCALE-KV exists to remove that prefill and cut first-token latency. With the refined base-map fit it stays close to the ceiling—no panel is significantly below native 14B and the LV-Eval gap shrinks from 7.45 to 3.44 F1—while reaching the first token up to 6.54× faster.

Native 14B is the ceiling, not the competitor. SCALE-KV lets the 14B answer without reading the document itself. Native 14B, which processes every document token, is therefore the reference SCALE-KV aims to approach while removing that cost; the question is how much of its quality is kept, and at what latency. At SCALE-KV's latency, the alternatives are smaller models: SCALE-KV exceeds native 8B, whose end-to-end first-token latency is on par, by +2.91 F1, and native 4B by +6.06 F1 on the seven-panel mean.

The refined base-map fit keeps SCALE-KV close to the ceiling. No panel is now significantly below native 14B. SCALE-KV is above native 14B on HotpotQA 8–16K (+0.34) and LongBench 8–16K (+1.32) and within 1.5 F1 on four further panels; on LV-Eval, the panel the review highlights (34.57 versus 42.02), the gap shrinks from 7.45 to 3.44 F1. Document-gain recovery reaches up to 101% on the six recovery panels: the 14B keeps the benefit of reading the document while its document prefill is removed.

At a fraction of the latency. In the same vLLM engine, with prefix caching enabled for native 14B, SCALE-KV reaches the first token up to 6.54× faster in the handoff setting and up to 1.53× faster for fresh requests, saving 1.4–3.2 s of first-token time per document.

Quality relative to native 14BFirst token relative to native 14B (vLLM, prefix caching)
Native 14Breference (reads the whole document)1×
SCALE-KV, handoffno panel significantly lower; recovery up to 101%up to 6.54× faster
SCALE-KV, fresh requestsame quality as the handoffup to 1.53× faster

Changes in the revision

  • A single definition of the source states, used consistently in Figures 1–2, Eq. (4) and the method description.
  • Table 1 with SCALE-KV's refined fit next to the original fit, and the panel-wise comparison with native 14B.
  • A discussion of the quality–latency trade-off that positions native 14B as the quality ceiling and the smaller native models as the latency-matched alternatives.

We hope these clarifications and results resolve your concerns, and we would be glad to provide further details during the discussion.

thanks for your clarification.

Thank you for your follow-up and for taking the time to consider our response. We wanted to kindly ask whether any concerns remain regarding the state definitions or the quality–latency tradeoff discussed in your review.

In particular, could you let us know whether there are any unresolved points that still affect your overall assessment? We would be happy to clarify them. If the additional evidence and explanations have addressed your main concerns, we would appreciate your reconsidering your assessment in light of these results.

Thank you again for your time and consideration.

Best regards,
The Authors

Reviewer_mAAYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose SCALE-KV for reusing a smaller model’s document states in a larger model. Specifically, a fixed base mapping is combined with low-rank corrections using source KV caches and hidden states. The converter and target-reader LoRA are jointly trained with generation objectives. Experiments on long-document QA show quality improvements over native source-model inference and reduced first-token latency compared with native target prefilling.

Critical strengths
  1. The motivation is practical. Constructing a reusable target prefix from source computation provides a useful alternative to repeating full-document prefilling.

  2. The experiments include capacity-matched controls, paired confidence intervals, and separate fresh-request and ready-state timing. The reported results demonstrate a useful quality–latency trade-off in the evaluated setting.

Critical weaknesses
  1. The contribution of state augmentation is not sufficiently supported. In Table 4, SCALE-KV obtains 47.72 Macro F1, compared with 47.73 for capacity-matched KV-only conversion and 48.46 for linear mapping with reader LoRA. The corresponding intervals in Table C2 include zero. These results do not establish that hidden-state inputs improve the cache interface. The smaller trainable parameter count is useful, but a direct latency and memory comparison with A1/A2 is needed to demonstrate its practical advantage.

  2. The efficiency comparisons need stronger controls. In the multi-question experiment, SCALE-KV reuses its converted prefix while native 14B repeats document prefilling for every question. Native 14B should also retain its prefix. Besides, the handoff results assume resident source KV and hidden states, and timing is limited to single-request HF/SDPA execution. These measurements support the stated execution path, but leave deployment benefits uncertain.

  3. The baseline coverage does not fully separate cross-scale reuse from task adaptation. Native 4B is not fine-tuned, whereas SCALE-KV receives task-specific supervision. A source model fine-tuned on the same data and an intermediate-size native model would provide useful quality–latency references. The LV-Eval result, where SCALE-KV falls 4.00 F1 below native 4B, also warrants further analysis.

  4. The checkpoint-selection procedure is unclear. Several recipes and capacity settings are compared on the evaluation panels, but the manuscript does not clearly describe the validation criterion used to select the main configuration. The reported bootstrap intervals quantify evaluation-sample uncertainty for fixed checkpoints; they do not address training-seed variability.

Questions for authors
  • Is there a reproducible setting where hidden-state augmentation improves over capacity-matched KV-only conversion?

  • How does SCALE-KV compare with native-14B prefix caching, a task-adapted 4B model, and native 8B inference?

  • How were the main recipe and checkpoint selected independently of the reported test results?

5 comments

Thank you for the detailed and constructive review, and for recognizing the practical motivation, the capacity-matched controls, the paired intervals and the separate fresh-request and ready-state timing. Following your suggestions, we ran these new experiments:

  • a matched-capacity setting in which the hidden state receives a full-width path (W1 / Q1);
  • A2 retrained under the main recipe with three seeds on both sides, and first-token latency for A1 and A2 (W1);
  • vLLM serving against native 14B with automatic prefix caching, batched throughput, and a 32B receiver (W2 / Q2);
  • a same-data adapted 4B, native 8B on all seven panels, and a document-mismatch control (W3 / Q2);
  • three training seeds for SCALE-KV and A2 (W4).

The table after the next two paragraphs maps each of your points to our answer.

Where SCALE-KV is used. SCALE-KV lets a large model take over a small model's reading of a long document, removing the large model's own document prefill. This handoff already occurs in deployed systems, where a lighter model processes the content before a stronger one answers:

  • Heterogeneous multi-agent systems. Lighter subagents search and read, and a stronger lead agent answers (e.g., Anthropic's multi-agent research system, with Claude Sonnet 4 subagents and a Claude Opus 4 lead; Chain of Agents, NeurIPS 2024). Today these systems hand over text; a 1,024-token text handoff falls far below SCALE-KV on every panel of Table 1.
  • Fast/slow routing and escalation. A fast model has already read the conversation or document when a stronger model takes over (the GPT-5 router in ChatGPT; RouteLLM, ICLR 2025; FrugalGPT cascades).
  • Documents processed before questions arrive. Precomputing document caches ahead of queries is established practice (TurboRAG, EMNLP 2025; CacheBlend, EuroSys 2025). SCALE-KV lets the small model do this step and hands the result to the large model when a question comes.

In all three, the large model has not read the document, so its own prefix cache cannot help. SCALE-KV removes exactly this prefill: 1.4–3.2 s per document in our vLLM measurements, with the first token up to 6.54× faster than native 14B with prefix caching.

A better fit of the frozen base map; the method is unchanged. We refined how SCALE-KV's frozen base key map is fitted (W1 below). Nothing else in the method changes: the same components, the same 40.7M trainable parameters, training data, recipe and step-900 checkpoint, and latency within 1%. The revised Table 1 will report SCALE-KV with the refined fit, next to the original fit. SCALE-KV quality numbers below use the refined fit and, unless marked otherwise, the pre-specified training seed (20260920); the A2 reference is averaged over three training seeds.

Summary of our responses

Your concernOur answerKey evidence
W1 / Q1: Do hidden-state inputs help at matched capacity?H_s is the more informative input; a full-width path turns this into a task gain in the pre-specified run.Probes and closed-form fits favor H_s on every layer; +1.96 F1 over a capacity-matched control.
W1: Quality versus A2SCALE-KV matches main-recipe A2 with the original fit and exceeds it with the refined fit.+0.04 F1 (three seeds each); +3.19 F1, significantly better on four panels; every seed significantly above A2.
W1: Latency and memory versus A1/A2The parameter saving is a training-memory saving; first-token latency is on the same level as A1/A2.81M versus 215M optimizer-state values; fresh-request first-token speedups of 1.57–1.67× over native 14B for all three.
W2 / Q2: Prefix caching, resident states, servingThe saving holds in vLLM with prefix caching enabled for native 14B.Up to 6.54× (handoff) and 1.53× (fresh) faster first token; 1.4–3.2 s saved per document.
W3 / Q2: Task adaptation, native 8B, LV-EvalOn par with a same-data adapted 4B, above native 8B, LV-Eval gap to native 4B closed.−0.58 / +2.91 / +0.01 F1; a wrong document drops SCALE-KV to native 14B's wrong-document level.
W4 / Q3: Selection and training seedsFixed final checkpoints; the base-map coordinate was chosen on a disjoint tuning panel; the gains hold for every seed.Step 900 for all models; per-seed table under W4.

W1 / Q1 — Does hidden-state augmentation help, and is there a reproducible setting?

Answer. The hidden state is the more informative input in every representation-level comparison, and a full-width path turns this into a +1.96 F1 gain over a capacity-matched control in the pre-specified run.

H_s is the better predictor of the target cache in every comparison we ran:

Representation-level testResult
Linear probe of the 14B's keys (held-out R²)H_s 0.733 vs. source KV 0.721
Linear probe of the 14B's values (held-out R²)H_s 0.587 vs. source KV 0.564
Adding the hidden state four layers deeperexplains up to 14% of the residual variance
Closed-form map at exactly matched capacityH_s better on all 40 target layers and every tested rank

The probe advantage holds on every probed layer.

These results motivate giving the hidden state a direct, full-width path into the target representation. SCALE-KV routes H_s through a rank-64 residual on the value path; here we map the source hidden state to the target's attention input and use the target's frozen k_proj, k_norm and v_proj to construct the cache. The control replaces the hidden state with a fixed orthonormal embedding of [K_s; V_s] of the same width. The modules, parameter counts and training recipe are identical.

With the pre-specified training seed 20260920, the hidden-state route improves the seven-panel mean by +1.96 F1, with a positive difference on all seven panels. Across three training seeds, the mean gain is +0.80 F1, positive on six of seven panels and significant on FRAMES 16–32K (the other two seeds give +0.55 and −0.12).

W1 — A1/A2: quality, latency and memory

Answer. SCALE-KV matches main-recipe A2 with the original fit and exceeds it with the refined fit; the parameter saving is a training-memory saving, and first-token latency is on the same level as A1/A2.

We retrained A2 with SCALE-KV's main recipe and normalization and evaluated all seven panels, using three training seeds on both sides. Fitting the frozen base key map before the target key normalization adds no trainable parameters:

Trainable parametersOptimizer-state valuesSeven-panel mean vs. A2 (F1)Panels vs. A2
A2, main recipe (three seeds)104.9M215M——
SCALE-KV, original fit (three seeds)40.7M81M+0.04no significant difference
SCALE-KV, refined fit40.7M81M+3.19significantly better on HotpotQA 8–16K, LongBench 8–16K, HotpotQA 16–32K and LV-Eval; worse on none

SCALE-KV trains 39% of A2's parameters, so the parameter saving is a saving in training memory. The refined map adds no parameters and leaves latency unchanged (within 1%): the quality gain comes from a better conversion interface, not a larger training budget.

At inference, SCALE-KV, A1 and A2 reach the first token at the same level. Measured with the same harness on the paper's timing subsets, they are 1.64–1.65×, 1.66–1.67× and 1.57× faster than native 14B for fresh requests, and 5.36–6.25×, 5.59–6.76× and 4.55–5.45× faster in the handoff setting (A2 with its full-rank deltas applied unfolded).

W2 / Q2 — Efficiency controls: prefix caching, resident states and serving

Answer. Inside vLLM, with automatic prefix caching enabled for native 14B, SCALE-KV still reaches the first token up to 6.54× faster in the handoff setting and up to 1.53× faster for fresh requests.

We integrated SCALE-KV into vLLM 0.18 on an A800, injecting the converted states into the paged KV cache and serving the reader LoRA through vLLM. The baseline is native Qwen3-14B with automatic prefix caching enabled in the same engine. These measurements use the original fit; the refined fit changes latency by less than 1%. Time to first token is measured per request on the paper's timing subsets.

Timing subsetQuestionsHandoff speedupFresh-request speedupFirst-prefix time: native 14B → SCALE-KV
HotpotQA 8–16K3275.64×1.50×2.08 s → 0.37 s
FRAMES 8–16K595.38×1.50×1.76 s → 0.33 s
FRAMES 16–32K646.54×1.53×3.73 s → 0.57 s

The handoff starts with the source states already on the GPU. Fresh requests include the source computation.

Prefix caching preserves the saving. Native 14B must build the document prefix once before caching helps; SCALE-KV builds it from the source states, saving 1.7 / 1.4 / 3.2 s per document (last column). The converted prefix then lives in vLLM's paged cache and is hit by later questions like any cached prefix. The revised multi-question comparison will cache the document on both sides.

Resident source states. This is the handoff setting described at the top; even fresh requests, which pay for the source computation, are up to 1.53× faster.

Beyond single requests. On these long-document subsets, batching provides little amortization of native 14B prefill: its throughput changes by only 1.3% from batch 1 to 8 on HotpotQA and 0.1% from batch 1 to 4 on FRAMES 16–32K, and the 14B/4B throughput ratio is stable. The cost SCALE-KV removes therefore persists at batch sizes up to 8. The saving also grows with the receiver: in the same engine, native Qwen3-32B takes up to 2.41× as long as the 14B to read the same documents, and the estimated 4B→32B handoff speedup is up to 9.8×.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.