acceptodds
Under review as a conference paper at ICLR 2027

ACCT: Asymmetric Cross-LLM Context Transfer

Abstract

In real-world deployments, large language model (LLM) systems are increasingly evolving from single-model invocation toward heterogeneous deployments in which different models are dynamically invoked for inference. When the same context is reused across models, however, each newly selected target model typically has to process it again, repeatedly incurring substantial computational costs. Therefore, we propose ACCT (Asymmetric Cross-LLM Context Transfer), which targets context-state transfer between models of different scales within the same model family. ACCT transforms the internal representations produced by a source model when processing the context into target-native sparse KV states that can be directly consumed by the target model, thereby enabling cross-model reuse of context computation and decoupling context computation from downstream inference. ACCT consists of three lightweight components: the Cartographer identifies query-relevant context for each target layer and KV head; the Allocator assigns state capacity under a global budget; and the Compiler transforms selected source-side representations into sparse target-native KV states. The target model can therefore perform inference using the transferred states without reprocessing the original context. We evaluate ACCT in two complementary settings comprising five general-capability benchmarks and six highly context-dependent tasks. Experiments show that ACCT matches or exceeds the target model's full-context inference performance on selected tasks, with outcomes varying across model pairs and datasets. At a context length of approximately 19K tokens, ACCT's KV state conversion achieves a 6.5-21.2x speedup over target context prefill. These results demonstrate the feasibility of transferring context computation across models under a limited state budget, offering a new approach to context reuse and efficient inference in heterogeneous model systems.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Cross-LLM Context Transfer, Large Language Model Inference, KV Cache

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_yWJBOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

ACCT transfers context-dependent KV states between different-sized LLMs from the same model family. The goal is to reuse KV states computed by a source model so that the target model can answer subsequent queries without processing the shared context again.

Critical strengths

S1. The appendix includes useful implementation details, including training configurations, parameter counts, and component-level latency breakdowns. The paper also reports unsuccessful transfer settings rather than presenting only favorable cases.

Critical weaknesses

W1. The experimental comparison is mainly against Target Full and variants of ACCT itself. A simpler cross-model KV mapping baseline under the same retention budget is missing, making it difficult to isolate the contribution of the proposed framework. I am also concerned by the unusually large gains over Target Full on SearchQA and TriviaQA (+20.4 and +15.8 F1 for Qwen3 4B→8B), particularly when BoolQ, RACE, and WikiHop show degradation. Since the reported QA metric is token-level F1, differences in answer length or output style could partly explain these gains. Without exact-match accuracy, answer lengths, and precision/recall, I cannot determine whether ACCT actually improves context utilization.

W2. The evidence for generalization across model families is limited. Although six transfer directions are evaluated, the strongest results are concentrated in Qwen3. General-capability retention is only about 65–76% for Llama and Qwen2.5, and context-dependent retention falls to 63.4% in some settings. The individual task results are also concerning. For Qwen2.5 7B→32B, HellaSwag drops from 61.10 to 29.60, while WikiText2 perplexity increases from 5.83 to 26.24. The additional Qwen3 14B→4B experiment further exposes the difficulty of transferring to smaller target models. The paper does not sufficiently explain these failures or characterize the conditions under which KV transfer is reliable.

W3. I find the efficiency claims overstated relative to the end-to-end measurements. The headline 6.5–21.2× speedup covers KV conversion components, not the full computation required to obtain and transfer the source KV states. Appendix K reports only 1.16–3.35× speedups for smaller-to-larger transfers once source prefill is counted. Reverse-direction transfers are actually slower than native target prefill, with speedups of 0.20–0.50×. These measurements are disclosed, but the abstract and main evaluation place much greater emphasis on the component-level numbers. Since deployment efficiency is a central motivation, I would expect the end-to-end results to receive comparable prominence. The latency evaluation is also restricted to batch size 1 with limited repetitions, so its relevance to practical serving workloads remains uncertain.

Questions for authors

Q1. What explains the large improvements over Target Full on SearchQA and TriviaQA (+20.4 and +15.8 F1 for Qwen3 4B→8B)? Were the prompts, generation settings, answer normalization, and evaluation code identical? Please report mean output lengths, token-level precision/recall, and exact-match accuracy for both methods.

Q2. Why does ACCT underperform on BoolQ, RACE, and WikiHop while performing substantially better on SearchQA and TriviaQA? Is this related to the SearchQA-based selection supervision, or does it reflect a limitation of KV compilation? How do the authors explain the severe degradation observed in Qwen3 14B→4B?

Q3. How does ACCT compare with a simple linear or ridge-based KV mapping baseline at the same 25% retention budget? Without this comparison, it is difficult to assess how much of the performance comes from the proposed three-component design rather than basic cross-model KV conversion.

Q4. Given that Appendix K reports end-to-end slowdowns for reverse-direction transfers, what practical deployment scenario would make these transfers worthwhile? Please also clarify the number of latency repetitions, report variability, and explain how the component-level speedups relate to end-to-end serving performance.

9 comments
AuthorsOpenReview ↗

We sincerely thank you for your insightful comments — New experiments and analyses on evaluation consistency, transfer performance, linear mapping baselines, and end-to-end efficiency (Part 1/4)

We appreciate the opportunity to clarify the evaluation, generalization, and efficiency of ACCT.

(Question1) Are the large gains on SearchQA and TriviaQA caused by evaluation differences?

Both methods evaluate the same 1,000 examples on SearchQA and TriviaQA, using exactly the same prompt construction, decoding settings, answer normalization, and evaluation code; The only difference is how the KV states are processed. The following table reports F1, exact match EMEM, token-level precision/recall, and mean output length, all in Target Full→ACCT order:

Table 8. Token-level evaluation on SearchQA and TriviaQA in both Qwen3 directions: F1, EM, precision, recall, and mean output length, shown in Target Full → ACCT order; the same 1,000 examples, prompts, decoding settings, and evaluation code are used.

Direction / datasetF1EMPrecisionRecallMean output length
4B→8B SearchQA46.65→67.0732.10→56.4041.76→64.6885.06→79.8916.02→7.63
4B→8B TriviaQA50.48→66.3133.20→56.0044.37→63.7788.20→78.6412.36→6.57
8B→4B SearchQA38.56→50.3323.60→37.6033.11→46.5885.10→79.5119.44→14.39
8B→4B TriviaQA37.68→54.5118.20→43.0030.27→50.8687.13→77.4015.76→11.00

In the Qwen3 4B→8B direction, ACCT produces shorter outputs with substantially higher token-level precision but lower recall. This indicates that the F1 gains are accompanied by more concise and precise answers.

(Question2) What causes the performance differences across tasks, and the degradation in 14B→4B?

Training distribution and task differences

We believe that performance differences across tasks are related to the data distribution used for selection supervision, rather than being a limitation of KV compilation. To test this, we replaced the original training set with RACE and compared the systems under the same method. RACE training improves RACE, BoolQ, and WikiHop, while SearchQA declines substantially:

Table 9. Effect of the selection-supervision training distribution: original training versus RACE-only training, on six QA and reading-comprehension datasets (F1 or accuracy).

DatasetOriginal trainingRACE training
SearchQA F162.8844.92
TriviaQA F160.7562.80
BoolQ accuracy55.6066.00
RACE accuracy68.1079.00
WikiHop accuracy37.8043.50
PubMedQA accuracy27.2044.00

This supports the view that selection supervision changes context-position preferences and task generalization, indicating that the training distribution is an important source of task differences.

Qwen3 14B→4B degradation

We found that the performance in 14B→4B transfer can be improved by expanding the search range for source-layer correspondences in the Compiler’s V Bridge. Specifically, we search within ±8 layers of the depth-aligned source layer, refit the V Bridge on 500 training examples, and train the associated residual module from scratch. The table below reports the results.

Table 10. Qwen3 14B→4B transfer: Target Full, the original Compiler, and the revised V Bridge with residual, across QA, commonsense, knowledge, and language-modeling metrics.

DatasetTarget FullOriginal CompilerNew Bridge + residual
SearchQA F138.560.3143.43
TriviaQA F137.680.6639.92
BoolQ accuracy84.000.6067.00
RACE accuracy87.1013.8056.30
WikiHop accuracy45.800.6025.80
PubMedQA accuracy34.000.6052.00
HellaSwag acc_norm41.5024.8038.20
WinoGrande acc_norm50.4050.7051.00
MMLU accuracy63.6025.5064.00
PIQA acc_norm67.9052.7064.90
WikiText2 PPL ↓81.2998.3840.57

The revised layer correspondence substantially alleviates the original Compiler's generation collapse. This indicates that V Bridge layer correspondence is one of the main causes of the 14B→4B degradation.

AuthorsOpenReview ↗

We sincerely thank you for your insightful comments — New experiments and analyses on evaluation consistency, transfer performance, linear mapping baselines, and end-to-end efficiency (Part 2/4)

(Question3) How does ACCT compare with a simple linear or ridge-based KV mapping baseline at the same 25% retention budget?

To address the request for comparison with simple linear or Ridge-based KV mapping, we fit an independent linear KV map using Ridge regression (linear regression with L2 regularization) to transform source-model KV into the target-model KV. Under the same 25% retention budget as ACCT, we either retain a random 25% of positions (Random25) or the most recent 25% (Recent25) . We also report results when retaining all KV positions (Full100) . The results are shown below:

Qwen3 4B→8B

Table 11. Ridge linear KV-mapping baselines versus ACCT in Qwen3 4B→8B at the same 25% retention budget: Random25, Recent25, Full100, ACCT, and Target Full.

MetricRandom25Recent25Full100ACCTTarget Full
SearchQA F1.067.048.162.671.467
TriviaQA F1.158.119.290.663.505
BoolQ acc..644.640.653.619.863
RACE acc..304.292.275.635.876
WikiHop acc..108.104.093.365.496
PubMedQA acc..382.312.214.402.488
HellaSwag.422.422.414.414.416
Winogrande.492.492.492.511.571
MMLU.300.315.312.657.597
PIQA.658.658.658.722.727
WikiText2 PPL38.1939.0632.2925.9432.91

Qwen3 8B→4B

Table 12. Ridge linear KV-mapping baselines versus ACCT in Qwen3 8B→4B at the same 25% retention budget: Random25, Recent25, Full100, ACCT, and Target Full.

MetricRandom25Recent25Full100ACCTTarget Full
SearchQA F1.007.006.000.503.386
TriviaQA F1.006.005.002.545.377
BoolQ acc..438.470.000.602.840
RACE acc..242.227.314.663.871
WikiHop acc..011.023.000.325.458
PubMedQA acc..118.134.000.496.336
HellaSwag.361.361.357.396.415
Winogrande.508.508.499.512.504
MMLU.240.247.245.582.636
PIQA.623.623.623.695.679
WikiText2 PPL140.35149.59209.6898.9281.29

Llama3.2 3B→Llama3.1 8B

Table 13. Ridge linear KV-mapping baselines versus ACCT in Llama3.2 3B→Llama3.1 8B at the same 25% retention budget: Random25, Recent25, Full100, ACCT, and Target Full.

MetricRandom25Recent25Full100ACCTTarget Full
SearchQA F1~0~0~0.669.634
TriviaQA F1~0~0~0.697.787
BoolQ acc..036.033.063.264.820
RACE acc..011.001.133.462.683
WikiHop acc..000.002.000.186.379
PubMedQA acc..038.022.044.404.686
HellaSwag.235.235.251.268.726
Winogrande.481.481.481.499.483
MMLU.235.233.241.419.586
PIQA.541.541.541.697.796
WikiText2 PPL3921.144705.13791.1664.917.21

Llama3.1 8B→Llama3.2 3B

Table 14. Ridge linear KV-mapping baselines versus ACCT in Llama3.1 8B→Llama3.2 3B at the same 25% retention budget: Random25, Recent25, Full100, ACCT, and Target Full.

MetricRandom25Recent25Full100ACCTTarget Full
SearchQA F1~0~0~0.540.380
TriviaQA F1~0~0~0.565.695
BoolQ acc..017.012.199.192.690
RACE acc..022.014.088.322.576
WikiHop acc..000.001.000.168.296
PubMedQA acc..000.000.008.322.660
HellaSwag.244.244.256.286.669
Winogrande.488.488.490.497.488
MMLU.262.244.235.330.452
PIQA.535.535.535.698.749
WikiText2 PPL831.52986.97376.2265.369.04

Except for BoolQ and HellaSwag in the Qwen3 4B→8B direction, ACCT is at least as good as the three Ridge linear-mapping baselines in the remaining 42 of 44 dataset–direction combinations. Even when the linear map is applied to all source KV positions, the simple linear transformation remains substantially behind ACCT, especially on QA tasks. This suggests that ACCT’s gains cannot be attributed to basic cross-model linear KV conversion or 25% token preservation alone, but are more likely due to the combination of selection, budget allocation, and target-space compilation.

AuthorsOpenReview ↗

We sincerely thank you for your insightful comments — New experiments and analyses on evaluation consistency, transfer performance, linear mapping baselines, and end-to-end efficiency (Part 3/4)

(Question4) When is reverse transfer useful? Clarify the number of latency repetitions, report variability, and explain how the component-level speedups relate to end-to-end serving performance.

Reverse transfer can save time after a model switch, with greater benefits for longer contexts. We consider a large model that has already processed the context before a later request switches to a smaller model. Timing starts from the existing source context state and ends when the target model produces its first token.

ACCT retains only a selected subset of the source KV states, so it carries over and attends over a much smaller KV cache. Fewer retained KV entries keep both transfer and attention costs low, and the advantage grows with context length.

To verify this, in the Qwen3 8B→4B reverse-transfer test, we compare complete time to first token (TTFT) for 4B ACCT, which receives 8B source states, against 4B Full. The results are as follows:

Table 15. Reverse-transfer end-to-end TTFT msms for Qwen3 8B→4B: 4B Full versus 8B→4B ACCT, with the Full/ACCT speedup ratio across context lengths.

Context tokensQwen3 4B Full TTFT msmsQwen3 8B→4B ACCT E2E TTFT msmsFull / ACCT
1,00166.912 ± 2.197177.208 ± 13.4540.378×
1,02466.606 ± 1.758161.873 ± 13.2130.411×
1,72998.808 ± 0.652161.924 ± 8.0300.610×
1,918125.013 ± 0.625198.558 ± 7.5240.630×
4,126256.981 ± 0.818248.501 ± 9.2551.034×
8,158500.611 ± 0.746286.763 ± 5.5081.746×
16,3891,170.165 ± 0.819494.767 ± 6.3372.365×
19,0701,442.491 ± 3.302554.810 ± 6.2402.600×
23,1371,897.978 ± 4.505676.419 ± 10.9602.806×
31,0612,852.636 ± 7.042867.859 ± 9.6193.287×
40,9604,370.560 ± 4.8321,118.101 ± 9.1813.909×
49,1525,901.704 ± 8.9441,329.043 ± 10.0344.441×
57,3447,618.111 ± 22.7321,546.903 ± 7.4854.925×
65,5369,398.035 ± 20.4911,777.037 ± 14.8075.289×
81,92013,888.522 ± 25.7322,217.335 ± 14.7346.264×
98,30419,430.680 ± 35.2932,680.459 ± 14.6757.249×
114,68825,766.647 ± 94.5093,124.194 ± 21.6048.247×
131,02031,281.868 ± 52.2313,536.288 ± 23.4618.846×

ACCT is slower than 4B Full at 1,001–1,918 tokens, becomes faster from 4,126 tokens, and reaches 8.846× at 131,020 context tokens. This shows that the reverse-transfer first-token latency advantage increases with context length.

Measurement variability. We performed 5 warm-up runs followed by 30 timed repetitions at each length. CV is the standard deviation divided by the mean and measures variation relative to runtime. The table below reports the mean, sample standard deviation, sample variance, and CV at 19,070 and 131,020 tokens for the 4B Full TTFT and 8B→4B ACCT E2E TTFT in the first table.

Table 16. TTFT measurement variability at 19,070 and 131,020 tokens: mean ± sample standard deviation, variance, and coefficient of variation CVCV, over 30 timed repetitions after 5 warm-up runs.

Context lengthMethodMean ± SD msmsVariance ms2ms²CV
19,0704B Full TTFT1442.491 ± 3.30210.9019590.229%
19,0704B ACCT E2E TTFT554.810 ± 6.24038.9397611.125%
131,0204B Full TTFT31281.868 ± 52.2312728.0840100.167%
131,0204B ACCT E2E TTFT3536.288 ± 23.461550.4112610.663%
Reviewer_oj76OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ACCT for transferring context states between different-sized LLMs within the same model family, so that the target model does not need to re-encode the full context. ACCT includes three components: the Cartographer selects important context tokens, the Allocator assigns the KV budget across target layers and heads, and the Compiler converts source representations into target KV states. The target model can then skip context prefill and only perform query encoding and subsequent generation. Experiments show that ACCT matches or outperforms Target Full on some model pairs and tasks, while reducing the cost of target-side context state construction.

Critical strengths
  1. ACCT enables direct conversion of source representations into target-native sparse KV states, eliminating the need for the target model to re-prefill the entire context after model switching.
  2. The evaluation covers bidirectional transfer across three model families, including both small-to-large and large-to-small settings, providing broader coverage of cross-model transfer scenarios.
Critical weaknesses
  1. ACCT does not provide a clean handoff from the source model to the target model. Each new query still requires source-side query processing, reselection, and recompilation, so the source model must remain available after the switch. This also raises an unresolved issue for multi-turn conversations: target-generated responses need to be incorporated into the source-side context; otherwise, subsequent query-dependent selection is performed over an incomplete conversation history and may select inappropriate context information.
  2. ACCT introduces several non-trivial architectural choices, but the current ablation study does not sufficiently justify these design choices. In particular, the Cartographer contains multiple components, while the paper mainly compares the full design against random selection. This mainly demonstrates the benefit of learned selection over random selection, but does not clarify the contribution of the individual design choices within the Cartographer.
Questions for authors
  1. How does ACCT handle multi-turn conversations and continued dependence on the source model? Since each new query requires source-side query computation, reselection, and recompilation, the source model must remain available even after switching to the target model. How are responses generated by the target model incorporated back into the source-side context? Without such synchronization, subsequent query-dependent selection may be based on an incomplete conversation history.
  2. What motivated the choice of the current benchmark suite? Since the transfer interface is trained only on SearchQA and some of the strongest gains are observed on open-domain QA tasks such as SearchQA and TriviaQA, do the authors have results on standard long-context benchmarks or other task types to further evaluate the generalization of ACCT?
6 comments
AuthorsOpenReview ↗

We sincerely thank you for your insightful comments — New experiments and analyses on Cartographer components, multi-turn dialogue, and long-context tasks (Part 1/2)

Thank you for your careful and constructive review.

(Weakness2) Insufficient ablations on the Cartographer’s components

We add four internal ablations in both Qwen3 4B→8B and 8B→4B directions: No FiLM removes FiLM conditioning; No Position removes explicit relative-position features while retaining target layer/head conditioning; No Matching removes the explicit token–evidence matching branch and its gate; and Mean Query replaces learned multi-slot aggregation with masked query mean pooling. Each variant retrains the Cartographer, Allocator, and Compiler, and evaluate on the datasets used in the paper. We refer to each evaluated method as a configuration.

To aggregate performance across tasks and metrics, on each dataset, we normalize each evaluation result relative to the best result among all evaluated configurations. For F1 and accuracy, we divide each configuration's score by the maximum score for that dataset. For WikiText2 PPL, where lower value indicate better performance, we divide the minimum PPL by each configuration's PPL. The best configuration for each dataset therefore receives a normalized score of 1. We the compuate the agrithmic mean across datasets. Results are shown below.

Qwen3 4B→8B (Average normalized score over 10 datasets):

Table 4. Cartographer ablation in Qwen3 4B→8B: mean normalized score over ten datasets, and cross-task variability σ, for full ACCT and each removed-component variant.

ConfigurationMean scoreCross-task variability σ
ACCT0.97460.0254
No FiLM0.89760.1655
No Position0.95590.0323
No Matching0.90310.1809
Mean Query0.88510.2049

Qwen3 8B→4B**(Average normalized score over 10 datasets):**

Table 5. Cartographer ablation in Qwen3 8B→4B: mean normalized score over ten datasets, and cross-task variability σ, for full ACCT and each removed-component variant.

ConfigurationMean scoreCross-task variability σ
ACCT0.91690.1342
No FiLM0.88970.1615
No Position0.88370.1112
No Matching0.89670.1773
Mean Query0.88120.1221

ACCT achieves the highest mean score among all configurations in both directions. This indicates that combining these components is necessary for the Cartographer.

(Question1) How does ACCT handle multi-turn conversations and continued dependence on the source model?

The source model remains available in subsequent turns. Specifically, we can use the following steps for multi-turn conversations:

  1. The source model encodes the complete text history so far and performs query-conditioned selection and KV compilation for the current user query.

  2. The target model loads the compiled KV states, performs native computation for the current query and subsequent generation, and produces a response.

  3. Select and generate complied KV for the current user query and the response actually generated by the target to be consumed in the following turns.

Using existing checkpoints, we evaluated this procedure on EmpatheticDialogues, MultiDoc2Dial, MultiWOZ, and DailyDialog, with 200 conversations per dataset. Each group reports ACCT, Target Full, and recovery rate.

Table 6. Multi-turn dialogue results on EmpatheticDialogues, MultiDoc2Dial, MultiWOZ, and DailyDialog (200 conversations per dataset), reported as ACCT, Target Full, and recovery rate across the four transfer directions.

DatasetQwen3 4B→8BQwen3 8B→4BLlama 3B→8BLlama 8B→3B
ACCTFullRecoveryACCTFullRecoveryACCTFullRecoveryACCTFullRecovery
EmpatheticDialogues6.5710.4962.6%10.3710.14102.3%4.637.4562.1%4.956.3977.5%
MultiDoc2Dial18.8025.6573.3%19.3124.1979.8%13.3121.6261.6%15.2916.2694.0%
MultiWOZ10.8017.8260.6%18.2318.7697.2%7.8311.2569.6%6.8710.1467.8%
DailyDialog6.9411.7059.3%11.7611.42103.0%5.827.6676.0%6.007.9875.2%

These datasets cover empathetic, document-grounded, task-oriented, and everyday dialogue. ACCT recovers part of the performance of Target Full across multiple tasks and transfer directions, showing that writing back the actual generated history and recompiling it basically supports subsequent dialogue turns.

AuthorsOpenReview ↗

We sincerely thank you for your insightful comments — New experiments and analyses on Cartographer components, multi-turn dialogue, and long-context tasks (Part 2/2)

(Question2) What motivated the benchmark suite, and are there standard long-context or other-task results?

We use two complementary benchmark groups. HellaSwag, Winogrande, MMLU, PIQA, and WikiText2 test general capabilities, including commonsense, knowledge/reasoning, and language modeling. SearchQA, TriviaQA, BoolQ, RACE, WikiHop, and PubMedQA test context use in open-domain QA, reading comprehension, binary decisions, multi-hop reasoning, and domain-specific QA. These are standard benchmarks with established task definitions and evaluation protocols, and we use task-appropriate metrics such as accuracy, F1, and perplexity. The suite therefore covers general ability, context utilization, and reasoning; These datasets are widely used in related work and provide an effective measure of method capability [1][2].

As suggested by the reviewer, we additionally evaluate the performance of our method on long-context and non-QA tasks. Specifically, we evaluated QMSum, RepoBench-P, SummScreenFD, and MeetingBank in four transfer directions. QMSum, SummScreenFD, and RepoBench-P are LongBench tasks covering long-meeting summarization, long-subtitle plot summarization, and cross-file code completion; MeetingBank adds real meeting-transcript summarization. For testing, we used fixed held-out subsets of 150 examples per direction for QMSum, 300 for RepoBench-P, and 200 for SummScreenFD and MeetingBank.

Table 7. Long-context and non-QA results on QMSum, RepoBench-P, SummScreenFD, and MeetingBank, reported as ACCT, Target Full, and recovery rate across the four transfer directions.

Dataset (metric)Qwen3 4B→8BQwen3 8B→4BLlama 3B→8BLlama 8B→3B
ACCTFullRecoveryACCTFullRecoveryACCTFullRecoveryACCTFullRecovery
QMSum (ROUGE)↑19.0724.1579.0%20.5924.5383.9%16.9424.9667.9%17.3123.7972.8%
RepoBench-P (Edit Similarity)↑53.5062.7985.2%47.5559.4280.0%51.0366.4676.8%49.0263.9776.6%
SummScreenFD (ROUGE)↑15.7518.6384.5%14.6816.8587.1%9.9314.2070.0%3.105.9652.1%
MeetingBank (ROUGE)↑17.7211.33156.5%15.9713.95114.5%10.7114.7372.7%7.0311.4961.2%

These results show that ACCT can recover a meaningful portion of the target model's capabilities in long-context tasks, providing evidence of its ability beyond QA tasks.

Thank you for the detailed response and additional experiments. The new results address some of my earlier concerns, especially regarding the Cartographer ablations. I have a few remaining questions.

1. Performance–Efficiency Trade-off in Multi-Turn Conversations

Under the proposed multi-turn procedure, the source model remains involved after switching: it must process responses generated by the target model, and each new query requires another round of query-conditioned selection and KV compilation.

This introduces additional computation and state maintenance overhead. Retaining source KV caches and relevant hidden states requires additional memory, while recomputing these states incurs further computational costs. It is unclear whether source states are incrementally updated or recomputed in the new multi-turn experiments, and how much overhead this introduces.

At the same time, Table 6 shows that 12 out of 16 results have recovery rates below 80%, with several around 60% of Target Full's performance.

Given these performance losses, my main concern is whether the savings from avoiding target prefill justify the additional source-side computation and state maintenance costs. The rebuttal does not report end-to-end latency or peak memory usage in multi-turn settings. Comparisons with native target-model inference under consistent conditions, including standard KV-cache reuse, would provide a clearer picture of the overall trade-off.

2. Benchmark Selection and Long-Context Generalization

The additional experiments on QMSum, RepoBench-P, SummScreenFD, and MeetingBank broaden the evaluation beyond QA tasks and help address my earlier concern.

However, the evaluation still relies on a relatively small set of selected tasks. A more comprehensive evaluation covering diverse long-context task categories would provide a more systematic assessment of ACCT's generalization capabilities. It would also be helpful to understand the motivation for selecting these particular tasks.

Additionally, clarifying how the evaluation subsets were selected and reporting the actual context lengths used in these experiments would help readers better interpret the results.

3. Component Ablations

The additional fine-grained Cartographer ablations largely address my earlier concern about component contributions. While these results provide evidence of the components' contributions to performance, it would also be useful to understand their computational overhead and whether the performance gains justify the added complexity.

Overall, I appreciate the additional experiments and clarifications. However, the practical performance–efficiency trade-off in multi-turn settings remains my main concern.

Reviewer_694ZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ACCT, a method for transferring context states between LLMs of different sizes within the same model family. It selects query-relevant context, allocates cache capacity across target layers and heads, and converts source representations into sparse target KV states, allowing the target to skip context prefill. Experiments across three model families show that retaining roughly 25% of the target KV slots can match or exceed full-context performance on some tasks, while causing substantial degradation on others.

Critical strengths
  • The idea of reusing context computation across models is interesting and has a practical use when a system switches between models of different sizes.
  • The paper is well organized and easy to follow.
  • The experiments cover several model families and both transfer directions.
Critical weaknesses
  • The benefit of learned allocation is less clear. Uniform allocation performs better on several tasks in the Qwen3 8B → 4B setting, so the results do not consistently support this design choice
Questions for authors

Please see the weakness.

2 comments
AuthorsOpenReview ↗

We sincerely thank you for your insightful comments — New experiments and analyses on improving allocator adaptivity in the Qwen3 8B→4B setting

(Weakness) About uniform allocation in the Qwen3 8B → 4B setting

Thank you for carefully reviewing our work. We agree with the reviewer that uniform allocation performs better on several tasks than Allocator in the Qwen3 8B→4B direction.

The original Allocator uses a fixed budget for each layer-head pair, which limits its ability to adapt to different queries and contexts. To mitigate this issue, we add a simple normalizer: the Cartographer scores are normalized by their mean across layer-heads, and the resulting normalized scores are added as a bias on the Allocator's original budgets. The experiment is conducted in the Qwen3 8B→4B direction. The results are below.

Table 3. Allocator ablation in the Qwen3 8B→4B direction: Full ACCT (reference), ACCT with the Cartographer-score normalizer, Uniform allocation, and the original ACCT, across the evaluation datasets (F1, accuracy, or PPL).

DatasetFull ACCT ref.ref.ACCT + normalizerUniformOriginal ACCT 8B→4B8B→4B
SearchQA F1 ↑0.6710.5610.4360.503
TriviaQA F1 ↑0.6630.5880.5060.545
RACE accuracy ↑0.6350.6910.6640.663
WikiHop accuracy ↑0.3650.3750.3590.325
PubMedQA accuracy ↑0.4020.3820.3360.496
HellaSwag accuracy ↑0.4140.4180.4000.369
WinoGrande accuracy ↑0.5110.5220.5070.513
MMLU accuracy ↑0.6570.6430.6430.582
PIQA accuracy ↑0.7220.7200.7320.690
WikiText2 PPL ↓25.9426.7229.2098.92

With the normalizer, ACCT outperforms Uniform and Original ACCT on SearchQA, TriviaQA, RACE, WikiHop, HellaSwag, and WinoGrande, suggesting that this simple calibration helps improve cross-dataset generalization.

AuthorsOpenReview ↗

Thanks for your comments

Dear reviewer:

We are writing to express our sincere gratitude for the time, effort, and expertise you dedicated to reviewing our paper. Your helpful comments do contribute to our work.

Reviewer_C423OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

ACCT tacklest the problem of KV-cache trasnfer between models, an extremely important problem with both memory and runtime performance implications. It accomplishes through a three tiered approach: (1) a cartographer that determines which KV-cache entries are important and should be retained, (2) an allocator that determines the capacity of each k/v head, and (3) a compiler that converts the KV-cache from one model family to another. The paper has an extensive evaluation across both performance (figure 2) as well as quality across LLama and Qwen3 model families.

Critical strengths

The paper tackles a timely problem. Moreover, the performance results are exceptionally good, it looks like the method can accelerate up to over 20x on certain tasks. Interestingly, looking at the trends the speedups accentuate with greater contexts, so this technique is scalable to long contexts as well.

The paper also has an extensive evaluation on the impact of model quality, tables 1, 2 and 3.

Critical weaknesses

The quality results, particularly table 1, tell an interesting story. For Language modelling, it seems like the proposed technique really hurts quality, with certain transfers reducing quality up to 10% of what it initially was. I appreciate the authors for putting these results in the paper, however, as it stands, these claims need to be justified and explained. I.e. why is this technique not suitable for language modeling? How can the authors plug this gap, since an almost 10x increase in perplexity for language modeling is quite significant (particularly given the importance of this task). Moreover, this occurs for 4/6 tried transfers.

Questions for authors
  1. Explain why the proposed method particularly impacts language modeling adversarially.
  2. What are some of the possible remedies to patch up language modeling scores to be within some reasonable tolerance. Typically since the larger the model, the lower the perplexity, which is why these results are particularly adversarial, since transferring KV-states to a larger model increases perplexity by 10x in certain scenarios (e.g. Llama 3B -> 8B).
2 comments
AuthorsOpenReview ↗

We sincerely thank you for your insightful comments — New experiments and analyses on language-modeling degradation and recovery

Thank you very much for pointing out the degradation in language-modeling performance. We address your two questions below.

(Question 1) Why is the language-modeling degradation so large?

We interpret your concern about language-modeling ability as a concern about the performance collapse on WikiText2 for the Llama and Qwen2.5 models. The main cause is hyperparameter choice.

This hyperparameter retains the most recent 32 tokens. WikiText2 features relatively short contexts. At a 25% retention rate, most of the budget for short examples is therefore unavailable for selecting earlier context. When the number of context tokens is <128, there is no budget at all for earlier context. The language-modeling capability of the Llama and Qwen2.5 models is weaker than that of Qwen3, the resulting degradation is more severe when earlier context cannot be retained. Diagnostic results on the original WikiText2 data in the two Llama transfer experiments show that the current ACCT hyperparameter configuration degrades sharply for contexts of at most 128 tokens, whereas Full KV PPL remains relatively stable:

Table 1. WikiText2 diagnostic perplexity PPLPPL for Full KV and ACCT in the two Llama transfer directions, split by context length ≤128vs 128tokens≤128 vs \>128 tokens on the original WikiText2 data.

Transfer directionContext lengthFull KV PPLACCT PPL
Llama 3B→8B≤1287.565567.89
Llama 3B→8B>1287.1424.85
Llama 8B→3B≤1289.49867.49
Llama 8B→3B>1288.9437.42

(Question 2) How can language-modeling performance be brought back to a reasonable range?

To address the lack of budget for selecting earlier context, we reduce the forced tail from 32 to 4 tokens. We conducted experiments with the Llama models on WikiText2, evaluating 500 examples per transfer direction. The results are shown in the table below. We can see that this change can improve the performance of ACCT, reducing PPL from 64.91 and 65.36 to 18.83 and 26.53, respectively. Here we retain 25% tokens, the so called 4x compression setting. Covering all context positions (1x ACCT) further reduces PPL to 17.98 and 21.72.

Table 2. PPL for Target Full, original ACCT, short-tail ACCT, and 1x ACCT under the original WikiText2 evaluation protocol. PPL is the exponential of the average negative log-likelihood, and the two methods incur 1.02 and 0.90 additional nats of cross-entropy compared to Target Full, respectively.

Transfer directionTarget FullOriginal ACCT (tail=32)Short-tail ACCT (tail=4)1x ACCT
Llama 3B→8B7.2164.9118.8317.98
Llama 8B→3B9.0465.3626.5321.72
AuthorsOpenReview ↗

A Gentle Follow-up on Rebuttal

Dear reviewer:

Thank you very much for your insightful and constructive comments. As the review deadline is approaching, we want to kindly check if there are any remaining concerns or points that require further clarification from our side.

Thank you for paying time and effort in reviewing our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.