- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how to transfer a larger model’s KV cache to a smaller model while preserving input-dependent information. Through cache-replacement experiments and update analysis, the authors show that a substantial portion of C2C’s gains persists without the corresponding source cache. They propose CACHET, which jointly processes source keys and values across heads, introduces a K/V-aware mean–residual alignment objective, and retains alignment during task training. Experiments show improved multiple-choice accuracy and successful transfer of passage information unavailable to the receiver.
Critical strengths
- Useful analysis of source dependence. The cache-replacement controls distinguish accuracy improvements from reliance on the sharer’s current input. The counterfactual passage experiments provide complementary evidence that the transferred cache enables the receiver to answer questions about unseen passages.
- Substantial empirical support for the design choices. The paper evaluates multiple model pairs and random seeds, and includes controlled studies of alignment retention, residual weighting, and translator architecture. The parameter-matched separated-translator comparison is particularly useful for assessing the benefit of joint source processing.
Critical weaknesses
- The methodological advance over DLC appears incremental. Receiver-native cache reconstruction, source-only translation, and two-stage training are already present in closely related work. CACHET’s contribution primarily consists of architectural and objective refinements. Although the ablations support these refinements, DLC-SI and the plain-alignment variant already succeed on the counterfactual passage task.
- The evaluation leaves practical generation capabilities unclear. The reported tasks use multiple-choice scoring through answer-token logits after a forced answer prefix, rather than free-form generation. This establishes answer-selection performance but does not establish whether transferred caches support coherent explanations, faithful open-ended answers, or sustained generation. Consequently, the practical scope remains limited.
Questions for authors
- Free-form generation: Can the authors compare option-logit scoring with freely generated answers on the same evaluation items? Reporting answer accuracy, format compliance, and explanation consistency would help establish whether the gains extend beyond constrained answer selection.
- Scaling and layer mapping: What explains the accuracy decline when increasing the sharer from 14B to 32B? Could convergence diagnostics and reconstruction errors help identify whether the limitation comes from training budget, translator capacity, or the fixed layer mapping?
- Practical efficiency: How does CACHET compare with using the sharer directly in terms of both accuracy and end-to-end latency? The comparison with text relay is useful, but a sharer-only baseline would clarify the practical trade-off.
- Multi-agent applications: How well does CACHET work in a multi-agent task where agents have access to different documents and must combine their information to answer a question? A comparison with text-based communication in terms of task performance and end-to-end latency would help establish its practical value.
5 comments
Response to Reviewer ueWb (2/2): Generation, scaling and cost
W2/Q1. We evaluated generated answers on short-answer QA without a forced answer prefix. We agree that forced option scoring alone does not establish the accuracy, format compliance or explanation consistency of actual answer generation. We therefore kept the questions and options but removed the forced answer prefix on the same 2,764 items, using greedy generation capped at 64 tokens (3 seeds). After correcting termination training, generated-answer accuracy is 70.9 ± 3.5% under flexible extraction, close to the same model's forced accuracy of 72.93 ± 1.83%. Under this extraction rule, much of the short-answer selection performance carries over to generation.
This evaluation also exposed an omission in our training-target construction. The original targets lacked EOS, and the model often continued producing output after answering. Since 70.8% of targets were truncated at 64 tokens, we appended EOS only to the naturally ending 29.2% and retrained with the other Stage-2 settings unchanged.
Generation before and after the target correction (accuracy/termination %, mean ± seed SD):
| Condition | Forced accuracy | Free: original extraction | Free: flexible extraction | EOS termination |
|---|---|---|---|---|
| Before | 73.20 ± 0.93 | 57.75 ± 8.79 | 71.7 ± 1.5 | 19.9 ± 10.7 |
| EOS appended | 72.93 ± 1.83 | 55.9 ± 13.9 | 70.9 ± 3.5 | 99.6 ± 0.5 |
DLC-SI, plain alignment and CE-only (LR 2e-6) trained on the same targets also terminate in only 1.3%, 19.8% and 26.1% of cases, respectively; a receiver-native cache through the same injection path terminates in 99.9%.
The original extraction rule remains primary; flexible extraction is a post-hoc diagnostic accepting unambiguous forms such as "B. [option text]". The correction improves termination rather than generated-answer accuracy. EOS termination rises from 19.9% to 99.6%, and from 0% to 93–98% on the 256-token explanation test.
We assessed format and explanation consistency separately. Pre-fix extraction rates are 80.9% under the original rule and 97.1% under the flexible one. A blinded LLM-assisted audit of 40 pre-fix outputs finds explanations supporting the final choice in 25/29/29 cases across seeds, versus 34 for the source; this is not a human evaluation. We thus report short-answer selection and termination, with explanation quality limited to this small auxiliary audit.
Q2. The submitted runs used a common mapping rule and training budget without 32B-specific optimization. With the 0.6B receiver fixed, seven-seed accuracy is 80.84 ± 1.06% for the 14B source and 77.93 ± 3.45% for 32B. We investigated whether the fixed mapping from the 32B source's 64 layers to the 28-layer receiver limits performance. For the lowest and median seeds from the original seven, we held translator architecture fixed and changed only the map or update budget:
| 32B setting | Lowest seed | Median seed |
|---|---|---|
| Submitted: 28 evenly spaced layers | 71.95 | 79.21 |
| All 64 layers, adjacent-layer averages | 78.80 | 81.94 |
| Last 28 layers | 83.00 | 82.68 |
| Submitted map, 2× updates | 83.00 | 80.07 |
Both mapping changes and additional updates improve both seeds. The last-28 map reaches 83.00/82.68%, supporting the importance of mapping choice when using the larger source. This is an exploratory diagnostic on two seeds selected after the original results. Systematic mapping and training-budget optimization across model scales remains future work.
Q3. Handoff reduces fixed-length generation latency by up to 23.1%. At 256 output tokens, end-to-end latency is 5,022.1/6,533.9 ms (CACHET/source), a 23.1% reduction; at 64 tokens it is 1,277.9/1,475.9 ms, a 13.4% reduction. These measurements use two B200 GPUs at batch 1 (GR2-B). In the separate option-scoring task, CACHET achieves 73.20 ± 0.93% versus 78.37% for the source, or 93.4% of source accuracy. This does not measure the quality of the fixed-length outputs. Short scoring, naturally ending answers, memory and communication costs are reported in GR2-B.
Follow-up to Reviewer ueWb
Thank you again for your review. As the discussion period closes, we briefly point to what in the revision bears on your two main concerns.
Contribution relative to DLC (W1). On MMLU-Pro, removing any one design component costs 8–28 PGR points (Table 4, 4B→0.6B). From 61.3 for CACHET, PGR falls to 33.1 without joint K/V processing, 49.5 without the mean–residual objective, 53.1 without K/V-specific residual scales, and 49.4 without alignment retention even after tuning its learning rate. DLC-SI reaches 40.3.
Generation (W2/Q1). The receiver can generate from the transferred cache without a forced answer prefix. Outputs originally failed to terminate because our training targets lacked EOS; after correcting the targets, 99.6% of outputs stop on their own, with no loss in answer-selection accuracy (Appendix E).
We would be grateful to hear whether these address your concerns, and we are happy to answer any remaining questions.
Response to Reviewer ueWb (1/2): Contribution and scope
Thank you for recognizing the source-dependence analysis and architectural controls.
W1. CACHET connects the diagnosis of source-insensitive gains to a cache-alignment objective and validates the resulting design. We share the source-only interface, native reconstruction and two-stage training with DLC; the revision states this in §5. Our starting point is different: whether an interface's accuracy gain reflects use of the corresponding source input, or adaptation that persists without it. The common-pool Mean controls make this measurable: a constant prefix retains about 8 pp over the receiver alone, while the matched cache adds about 29 pp (R1-A). The receiver-native target then shows why total reconstruction error is not the right objective for the matched part: the batch-mean component accounts for 79.7% of K energy but 25.7% of V energy. CACHET therefore supervises mean and residual components separately, with separate K/V residual scales.
Three controls validate the design:
Against the strongest DLC setting we evaluated, CACHET scores 71.66% versus 66.91% on 4B→0.6B (7 seeds), and 77.97% (same-input) / 77.62% (mixed) versus 75.05% on 8B→1.7B (common seeds; R1-C). As you note, DLC-SI and plain alignment also pass the passage test; that test diagnoses input dependence, and the comparison between methods rests on these benchmarks and ablations.
Q4. Multi-agent tasks with different documents. Your proposed task would test evidence integration beyond our single-source experiments; combining distinct agents' documents remains untested. The new passage-only ranking (GR2-A) and two-turn prefix appending (UVDt Q3) address prerequisites for that setting: context can be transferred without the source seeing the question, and extended across turns without reprocessing.