- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SCALE-KV, which lets a large target model answer from a KV cache converted from a small source model, so the target does not prefill the document itself. Each target layer reads a source layer chosen by relative depth. Keys are mapped by a frozen affine map with a low rank residual after removing RoPE.Values are mapped by a fixed QR basis and a trainable affine readout. A low rank residual that also takes the source hidden states as input is added to this mapping. The converter and a LoRA on the target attention are trained jointly with answer cross entropy and distillation from native 14B, while both backbones stay frozen.
Critical strengths
-
The cost accounting is end to end. Fresh request TTFT includes source prefill, conversion, and target suffix processing. Handoffs are reported separately, and a latency matched mixture of native 4B and 14B is given as a reference.
-
The results show that reading a converted cache benefits from adapting the target reader. Retraining without reader LoRA lowers accuracy, and mapping baselines without target side adaptation fall below native 4B.
Critical weaknesses
-
The proposed converter design is not supported by the paper's own controlled comparison. Adding source hidden states, the component the method is named after, gives no measurable gain at matched capacity. In the same controlled setting, a plain linear KV map with the same reader LoRA and losses (A2) performs at least as well. The advantage over A2 in Table 1 compares the main checkpoint, whose recipe is compared with alternatives only on the test panels, against A2 from the controlled runs. The parameter reduction applies only to trainable parameters. The evidence therefore supports linear mapping plus reader LoRA, and shows no advantage for the state augmented design.
-
The improvement over simple alternatives is limited or confounded. The native 4B reference receives no task adaptation, while SCALE-KV is trained with gold answers, and removing this supervision brings it close to native 4B on HotpotQA. SCALE-KV falls below native 4B on LV-Eval, and its gain over a latency matched random mix of 4B and 14B is inconsistent across panels. The multi question comparison also omits prefix caching for native 14B, and all latency measurements use batch size 1 without a serving framework.
Questions for authors
-
What data are the converter and reader trained on, and how was the main recipe selected?
-
Native 4B fine tuned on the same data and losses, evaluated on all panels.
-
A2 trained under the main recipe and normalization, compared with SCALE-KV across multiple seeds.
-
Latency comparison against native 14B with prefix caching in a serving framework such as vLLM or SGLang.
7 comments
W1 / Q3 — A2 under the main recipe, across seeds
Answer. Under the main recipe with three seeds on both sides, SCALE-KV matches A2 with the original fit and, with the refined fit, is significantly better than A2 for every seed.
As requested, we retrained A2 with SCALE-KV's main recipe and normalization and evaluated all seven panels, using three training seeds on both sides. With the base map as originally fitted, SCALE-KV already matches A2 (+0.04 F1 on the seven-panel mean, no panel significantly different) with 40.7M trainable parameters instead of 104.9M.
Fitting the frozen base key map before the target key normalization adds no trainable parameters and puts SCALE-KV +3.19 F1 above the same three-seed A2 reference, significantly better on HotpotQA 8–16K, LongBench 8–16K, HotpotQA 16–32K and LV-Eval and significantly worse on none. Every training seed reproduces this:
| SCALE-KV training seed | Seven-panel mean vs. A2 | Panels significantly better than A2 | Panels significantly worse than A2 | Seven-panel mean vs. original fit, same seed |
|---|---|---|---|---|
| 20260920 (pre-specified) | +3.19 | HP 8–16K, LB 8–16K, HP 16–32K, LV-Eval | none | +2.91 |
| 20261107 | +1.63 | the same four | none | +1.83 |
| 20261108 | +2.41 | the same four | none | +2.39 |
HP = HotpotQA, LB = LongBench; A2 is the three-seed reference. Every difference in the two mean columns is statistically significant.
W1 — Trainable parameters and deployment cost
Answer. The saving is in training memory (81M versus 215M optimizer-state values), and first-token latency is on the same level as A2.
| Trainable parameters | Optimizer-state values | First-token speedup over native 14B: fresh / handoff | Seven-panel mean vs. A2 (F1) | |
|---|---|---|---|---|
| A2, main recipe | 104.9M | 215M | 1.57× / 4.55–5.45× (deltas applied unfolded) | — |
| SCALE-KV, refined fit | 40.7M | 81M | 1.64–1.65× / 5.36–6.25× | +3.19 |
Both are timed with the same harness on the paper's timing subsets. The refined map adds no parameters and leaves latency unchanged (within 1%): the quality gain comes from a better conversion interface, not a larger training budget.
W1 — Hidden-state inputs at matched capacity
Answer. The hidden state is the more informative input, and a full-width path turns this into a +1.96 F1 gain over a capacity-matched control in the pre-specified run.
H_s is the better predictor of the target cache in every comparison we ran:
| Representation-level test | Result |
|---|---|
| Linear probe of the 14B's keys (held-out R²) | H_s 0.733 vs. source KV 0.721 |
| Linear probe of the 14B's values (held-out R²) | H_s 0.587 vs. source KV 0.564 |
| Closed-form map at exactly matched capacity | H_s better on all 40 target layers and every tested rank |
The probe advantage holds on every probed layer.
These results motivate giving the hidden state a direct, full-width path into the target representation. SCALE-KV routes H_s through a rank-64 residual on the value path; here we map the source hidden state to the target's attention input and use the target's frozen k_proj, k_norm and v_proj to construct the cache. The control replaces the hidden state with a fixed orthonormal embedding of [K_s; V_s] of the same width. The modules, parameter counts and training recipe are identical. With the pre-specified training seed 20260920, the hidden-state route improves the seven-panel mean by +1.96 F1, with a positive difference on all seven panels. Across three training seeds, the mean gain is +0.80 F1, positive on six of seven panels and significant on FRAMES 16–32K (the other two seeds give +0.55 and −0.12).
W1 / Q1 — How the recipe and checkpoint were selected
Answer. SCALE-KV and A2 now share the same recipe, normalization and checkpoint rule, and the coordinate of the refined base-map fit was chosen on a tuning panel disjoint from the test panels.
All models, including the main-recipe A2, use the fixed final checkpoint at step 900; no intermediate checkpoint is selected.
W2 / Q2 — Same-data adapted 4B, supervision and LV-Eval
Answer. SCALE-KV is on par with the same-data adapted 4B while keeping the 14B as the answering model, and its LV-Eval gap to native 4B is closed.
The content-mismatch control tests whether the 14B actually uses the transferred document. Replacing the correct document with a length-matched wrong one gives:
| Panel | SCALE-KV: correct document | SCALE-KV: wrong document | Native 14B: same wrong document |
|---|---|---|---|
| FRAMES 8–16K | 36.8 | 11.8 | 9.4 |
| HotpotQA 8–16K | 64.1 | 18.5 | 20.4 |
| LongBench | 47.1 | 14.5 | 12.8 |
With the wrong document, SCALE-KV falls to the level of native 14B reading that same wrong document (no significant difference on any panel). With the correct one, it recovers up to 101% of native 14B's document gain on the six recovery panels.
We fine-tuned the 4B with the same data, losses and LoRA setup and evaluated all seven panels. A fine-tuned 4B replaces the 14B; SCALE-KV keeps the 14B as the answering model, at comparable quality:
| Reference | SCALE-KV − reference (F1) |
|---|---|
| Same-data adapted 4B, seven-panel mean | −0.58 (on par) |
| Native 8B, seven-panel mean | +2.91 |
| Native 4B, seven-panel mean | +6.06 |
| Native 4B, LV-Eval (the review noted a deficit of about four points) | +0.01 |
Thank you for the precise review and the concrete requests, and for noting the end-to-end cost accounting and the role of reader adaptation. We ran each requested experiment:
We also added a matched-capacity full-width hidden-state experiment, first-token latency for A2, a document-mismatch control and a recomputed latency-matched mixture (W1, W2). The table after the next two paragraphs maps each of your points to our answer.
Where SCALE-KV is used. SCALE-KV lets a large model take over a small model's reading of a long document, removing the large model's own document prefill. This handoff already occurs in deployed systems, where a lighter model processes the content before a stronger one answers:
In all three, the large model has not read the document, so its own prefix cache cannot help. SCALE-KV removes exactly this prefill: 1.4–3.2 s per document in our vLLM measurements, with the first token up to 6.54× faster than native 14B with prefix caching.
A better fit of the frozen base map; the method is unchanged. We refined how SCALE-KV's frozen base key map is fitted (W1 / Q3 below). Nothing else in the method changes: the same components, the same 40.7M trainable parameters, training data, recipe and step-900 checkpoint, and latency within 1%. The revised Table 1 will report SCALE-KV with the refined fit, next to the original fit. SCALE-KV quality numbers below use the refined fit and, unless marked otherwise, the pre-specified training seed (20260920); the A2 reference is averaged over three training seeds.
Summary of our responses