- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2(+1)
- Soundness
- 2(+1)
Read review
Paper summary
ACCT transfers context-dependent KV states between different-sized LLMs from the same model family. The goal is to reuse KV states computed by a source model so that the target model can answer subsequent queries without processing the shared context again.
Critical strengths
S1. The appendix includes useful implementation details, including training configurations, parameter counts, and component-level latency breakdowns. The paper also reports unsuccessful transfer settings rather than presenting only favorable cases.
Critical weaknesses
W1. The experimental comparison is mainly against Target Full and variants of ACCT itself. A simpler cross-model KV mapping baseline under the same retention budget is missing, making it difficult to isolate the contribution of the proposed framework. I am also concerned by the unusually large gains over Target Full on SearchQA and TriviaQA (+20.4 and +15.8 F1 for Qwen3 4B→8B), particularly when BoolQ, RACE, and WikiHop show degradation. Since the reported QA metric is token-level F1, differences in answer length or output style could partly explain these gains. Without exact-match accuracy, answer lengths, and precision/recall, I cannot determine whether ACCT actually improves context utilization.
W2. The evidence for generalization across model families is limited. Although six transfer directions are evaluated, the strongest results are concentrated in Qwen3. General-capability retention is only about 65–76% for Llama and Qwen2.5, and context-dependent retention falls to 63.4% in some settings. The individual task results are also concerning. For Qwen2.5 7B→32B, HellaSwag drops from 61.10 to 29.60, while WikiText2 perplexity increases from 5.83 to 26.24. The additional Qwen3 14B→4B experiment further exposes the difficulty of transferring to smaller target models. The paper does not sufficiently explain these failures or characterize the conditions under which KV transfer is reliable.
W3. I find the efficiency claims overstated relative to the end-to-end measurements. The headline 6.5–21.2× speedup covers KV conversion components, not the full computation required to obtain and transfer the source KV states. Appendix K reports only 1.16–3.35× speedups for smaller-to-larger transfers once source prefill is counted. Reverse-direction transfers are actually slower than native target prefill, with speedups of 0.20–0.50×. These measurements are disclosed, but the abstract and main evaluation place much greater emphasis on the component-level numbers. Since deployment efficiency is a central motivation, I would expect the end-to-end results to receive comparable prominence. The latency evaluation is also restricted to batch size 1 with limited repetitions, so its relevance to practical serving workloads remains uncertain.
Questions for authors
Q1. What explains the large improvements over Target Full on SearchQA and TriviaQA (+20.4 and +15.8 F1 for Qwen3 4B→8B)? Were the prompts, generation settings, answer normalization, and evaluation code identical? Please report mean output lengths, token-level precision/recall, and exact-match accuracy for both methods.
Q2. Why does ACCT underperform on BoolQ, RACE, and WikiHop while performing substantially better on SearchQA and TriviaQA? Is this related to the SearchQA-based selection supervision, or does it reflect a limitation of KV compilation? How do the authors explain the severe degradation observed in Qwen3 14B→4B?
Q3. How does ACCT compare with a simple linear or ridge-based KV mapping baseline at the same 25% retention budget? Without this comparison, it is difficult to assess how much of the performance comes from the proposed three-component design rather than basic cross-model KV conversion.
Q4. Given that Appendix K reports end-to-end slowdowns for reverse-direction transfers, what practical deployment scenario would make these transfers worthwhile? Please also clarify the number of latency repetitions, report variability, and explain how the component-level speedups relate to end-to-end serving performance.
9 comments
We sincerely thank you for your insightful comments — New experiments and analyses on evaluation consistency, transfer performance, linear mapping baselines, and end-to-end efficiency (Part 2/4)
(Question3)
How does ACCT compare with a simple linear or ridge-based KV mapping baseline at the same 25% retention budget?
To address the request for comparison with simple linear or Ridge-based KV mapping, we fit an independent linear KV map using Ridge regression (linear regression with L2 regularization) to transform source-model KV into the target-model KV. Under the same 25% retention budget as ACCT, we either retain a random 25% of positions (Random25) or the most recent 25% (Recent25) . We also report results when retaining all KV positions (Full100) . The results are shown below:
Qwen3 4B→8B
Table 11. Ridge linear KV-mapping baselines versus ACCT in Qwen3 4B→8B at the same 25% retention budget: Random25, Recent25, Full100, ACCT, and Target Full.
| Metric | Random25 | Recent25 | Full100 | ACCT | Target Full |
|---|---|---|---|---|---|
| SearchQA F1 | .067 | .048 | .162 | .671 | .467 |
| TriviaQA F1 | .158 | .119 | .290 | .663 | .505 |
| BoolQ acc. | .644 | .640 | .653 | .619 | .863 |
| RACE acc. | .304 | .292 | .275 | .635 | .876 |
| WikiHop acc. | .108 | .104 | .093 | .365 | .496 |
| PubMedQA acc. | .382 | .312 | .214 | .402 | .488 |
| HellaSwag | .422 | .422 | .414 | .414 | .416 |
| Winogrande | .492 | .492 | .492 | .511 | .571 |
| MMLU | .300 | .315 | .312 | .657 | .597 |
| PIQA | .658 | .658 | .658 | .722 | .727 |
| WikiText2 PPL | 38.19 | 39.06 | 32.29 | 25.94 | 32.91 |
Qwen3 8B→4B
Table 12. Ridge linear KV-mapping baselines versus ACCT in Qwen3 8B→4B at the same 25% retention budget: Random25, Recent25, Full100, ACCT, and Target Full.
| Metric | Random25 | Recent25 | Full100 | ACCT | Target Full |
|---|---|---|---|---|---|
| SearchQA F1 | .007 | .006 | .000 | .503 | .386 |
| TriviaQA F1 | .006 | .005 | .002 | .545 | .377 |
| BoolQ acc. | .438 | .470 | .000 | .602 | .840 |
| RACE acc. | .242 | .227 | .314 | .663 | .871 |
| WikiHop acc. | .011 | .023 | .000 | .325 | .458 |
| PubMedQA acc. | .118 | .134 | .000 | .496 | .336 |
| HellaSwag | .361 | .361 | .357 | .396 | .415 |
| Winogrande | .508 | .508 | .499 | .512 | .504 |
| MMLU | .240 | .247 | .245 | .582 | .636 |
| PIQA | .623 | .623 | .623 | .695 | .679 |
| WikiText2 PPL | 140.35 | 149.59 | 209.68 | 98.92 | 81.29 |
Llama3.2 3B→Llama3.1 8B
Table 13. Ridge linear KV-mapping baselines versus ACCT in Llama3.2 3B→Llama3.1 8B at the same 25% retention budget: Random25, Recent25, Full100, ACCT, and Target Full.
| Metric | Random25 | Recent25 | Full100 | ACCT | Target Full |
|---|---|---|---|---|---|
| SearchQA F1 | ~0 | ~0 | ~0 | .669 | .634 |
| TriviaQA F1 | ~0 | ~0 | ~0 | .697 | .787 |
| BoolQ acc. | .036 | .033 | .063 | .264 | .820 |
| RACE acc. | .011 | .001 | .133 | .462 | .683 |
| WikiHop acc. | .000 | .002 | .000 | .186 | .379 |
| PubMedQA acc. | .038 | .022 | .044 | .404 | .686 |
| HellaSwag | .235 | .235 | .251 | .268 | .726 |
| Winogrande | .481 | .481 | .481 | .499 | .483 |
| MMLU | .235 | .233 | .241 | .419 | .586 |
| PIQA | .541 | .541 | .541 | .697 | .796 |
| WikiText2 PPL | 3921.14 | 4705.13 | 791.16 | 64.91 | 7.21 |
Llama3.1 8B→Llama3.2 3B
Table 14. Ridge linear KV-mapping baselines versus ACCT in Llama3.1 8B→Llama3.2 3B at the same 25% retention budget: Random25, Recent25, Full100, ACCT, and Target Full.
| Metric | Random25 | Recent25 | Full100 | ACCT | Target Full |
|---|---|---|---|---|---|
| SearchQA F1 | ~0 | ~0 | ~0 | .540 | .380 |
| TriviaQA F1 | ~0 | ~0 | ~0 | .565 | .695 |
| BoolQ acc. | .017 | .012 | .199 | .192 | .690 |
| RACE acc. | .022 | .014 | .088 | .322 | .576 |
| WikiHop acc. | .000 | .001 | .000 | .168 | .296 |
| PubMedQA acc. | .000 | .000 | .008 | .322 | .660 |
| HellaSwag | .244 | .244 | .256 | .286 | .669 |
| Winogrande | .488 | .488 | .490 | .497 | .488 |
| MMLU | .262 | .244 | .235 | .330 | .452 |
| PIQA | .535 | .535 | .535 | .698 | .749 |
| WikiText2 PPL | 831.52 | 986.97 | 376.22 | 65.36 | 9.04 |
Except for BoolQ and HellaSwag in the Qwen3 4B→8B direction, ACCT is at least as good as the three Ridge linear-mapping baselines in the remaining 42 of 44 dataset–direction combinations. Even when the linear map is applied to all source KV positions, the simple linear transformation remains substantially behind ACCT, especially on QA tasks. This suggests that ACCT’s gains cannot be attributed to basic cross-model linear KV conversion or 25% token preservation alone, but are more likely due to the combination of selection, budget allocation, and target-space compilation.
We sincerely thank you for your insightful comments — New experiments and analyses on evaluation consistency, transfer performance, linear mapping baselines, and end-to-end efficiency (Part 3/4)
(Question4)
When is reverse transfer useful? Clarify the number of latency repetitions, report variability, and explain how the component-level speedups relate to end-to-end serving performance.
Reverse transfer can save time after a model switch, with greater benefits for longer contexts. We consider a large model that has already processed the context before a later request switches to a smaller model. Timing starts from the existing source context state and ends when the target model produces its first token.
ACCT retains only a selected subset of the source KV states, so it carries over and attends over a much smaller KV cache. Fewer retained KV entries keep both transfer and attention costs low, and the advantage grows with context length.
To verify this, in the Qwen3 8B→4B reverse-transfer test, we compare complete time to first token (TTFT) for 4B ACCT, which receives 8B source states, against 4B Full. The results are as follows:
Table 15. Reverse-transfer end-to-end TTFT for Qwen3 8B→4B: 4B Full versus 8B→4B ACCT, with the Full/ACCT speedup ratio across context lengths.
| Context tokens | Qwen3 4B Full TTFT | Qwen3 8B→4B ACCT E2E TTFT | Full / ACCT |
|---|---|---|---|
| 1,001 | 66.912 ± 2.197 | 177.208 ± 13.454 | 0.378× |
| 1,024 | 66.606 ± 1.758 | 161.873 ± 13.213 | 0.411× |
| 1,729 | 98.808 ± 0.652 | 161.924 ± 8.030 | 0.610× |
| 1,918 | 125.013 ± 0.625 | 198.558 ± 7.524 | 0.630× |
| 4,126 | 256.981 ± 0.818 | 248.501 ± 9.255 | 1.034× |
| 8,158 | 500.611 ± 0.746 | 286.763 ± 5.508 | 1.746× |
| 16,389 | 1,170.165 ± 0.819 | 494.767 ± 6.337 | 2.365× |
| 19,070 | 1,442.491 ± 3.302 | 554.810 ± 6.240 | 2.600× |
| 23,137 | 1,897.978 ± 4.505 | 676.419 ± 10.960 | 2.806× |
| 31,061 | 2,852.636 ± 7.042 | 867.859 ± 9.619 | 3.287× |
| 40,960 | 4,370.560 ± 4.832 | 1,118.101 ± 9.181 | 3.909× |
| 49,152 | 5,901.704 ± 8.944 | 1,329.043 ± 10.034 | 4.441× |
| 57,344 | 7,618.111 ± 22.732 | 1,546.903 ± 7.485 | 4.925× |
| 65,536 | 9,398.035 ± 20.491 | 1,777.037 ± 14.807 | 5.289× |
| 81,920 | 13,888.522 ± 25.732 | 2,217.335 ± 14.734 | 6.264× |
| 98,304 | 19,430.680 ± 35.293 | 2,680.459 ± 14.675 | 7.249× |
| 114,688 | 25,766.647 ± 94.509 | 3,124.194 ± 21.604 | 8.247× |
| 131,020 | 31,281.868 ± 52.231 | 3,536.288 ± 23.461 | 8.846× |
ACCT is slower than 4B Full at 1,001–1,918 tokens, becomes faster from 4,126 tokens, and reaches 8.846× at 131,020 context tokens. This shows that the reverse-transfer first-token latency advantage increases with context length.
Measurement variability. We performed 5 warm-up runs followed by 30 timed repetitions at each length. CV is the standard deviation divided by the mean and measures variation relative to runtime. The table below reports the mean, sample standard deviation, sample variance, and CV at 19,070 and 131,020 tokens for the 4B Full TTFT and 8B→4B ACCT E2E TTFT in the first table.
Table 16. TTFT measurement variability at 19,070 and 131,020 tokens: mean ± sample standard deviation, variance, and coefficient of variation , over 30 timed repetitions after 5 warm-up runs.
| Context length | Method | Mean ± SD | Variance | CV |
|---|---|---|---|---|
| 19,070 | 4B Full TTFT | 1442.491 ± 3.302 | 10.901959 | 0.229% |
| 19,070 | 4B ACCT E2E TTFT | 554.810 ± 6.240 | 38.939761 | 1.125% |
| 131,020 | 4B Full TTFT | 31281.868 ± 52.231 | 2728.084010 | 0.167% |
| 131,020 | 4B ACCT E2E TTFT | 3536.288 ± 23.461 | 550.411261 | 0.663% |
We sincerely thank you for your insightful comments — New experiments and analyses on evaluation consistency, transfer performance, linear mapping baselines, and end-to-end efficiency (Part 1/4)
We appreciate the opportunity to clarify the evaluation, generalization, and efficiency of ACCT.
Both methods evaluate the same 1,000 examples on SearchQA and TriviaQA, using exactly the same prompt construction, decoding settings, answer normalization, and evaluation code; The only difference is how the KV states are processed. The following table reports F1, exact match EM, token-level precision/recall, and mean output length, all in Target Full→ACCT order:
Table 8. Token-level evaluation on SearchQA and TriviaQA in both Qwen3 directions: F1, EM, precision, recall, and mean output length, shown in Target Full → ACCT order; the same 1,000 examples, prompts, decoding settings, and evaluation code are used.
In the Qwen3 4B→8B direction, ACCT produces shorter outputs with substantially higher token-level precision but lower recall. This indicates that the F1 gains are accompanied by more concise and precise answers.
Training distribution and task differences
We believe that performance differences across tasks are related to the data distribution used for selection supervision, rather than being a limitation of KV compilation. To test this, we replaced the original training set with RACE and compared the systems under the same method. RACE training improves RACE, BoolQ, and WikiHop, while SearchQA declines substantially:
Table 9. Effect of the selection-supervision training distribution: original training versus RACE-only training, on six QA and reading-comprehension datasets (F1 or accuracy).
This supports the view that selection supervision changes context-position preferences and task generalization, indicating that the training distribution is an important source of task differences.
Qwen3 14B→4B degradation
We found that the performance in 14B→4B transfer can be improved by expanding the search range for source-layer correspondences in the Compiler’s V Bridge. Specifically, we search within ±8 layers of the depth-aligned source layer, refit the V Bridge on 500 training examples, and train the associated residual module from scratch. The table below reports the results.
Table 10. Qwen3 14B→4B transfer: Target Full, the original Compiler, and the revised V Bridge with residual, across QA, commonsense, knowledge, and language-modeling metrics.
The revised layer correspondence substantially alleviates the original Compiler's generation collapse. This indicates that V Bridge layer correspondence is one of the main causes of the 14B→4B degradation.