ACCT: Asymmetric Cross-LLM Context Transfer

In real-world deployments, large language model (LLM) systems are increasingly evolving from single-model invocation toward heterogeneous deployments in which different models are dynamically invoked for inference. When the same context is reused across models, however, each newly selected target model typically has to process it again, repeatedly incurring substantial computational costs. Therefore, we propose ACCT (Asymmetric Cross-LLM Context Transfer), which targets context-state transfer between models of different scales within the same model family. ACCT transforms the internal representations produced by a source model when processing the context into target-native sparse KV states that can be directly consumed by the target model, thereby enabling cross-model reuse of context computation and decoupling context computation from downstream inference. ACCT consists of three lightweight components: the Cartographer identifies query-relevant context for each target layer and KV head; the Allocator assigns state capacity under a global budget; and the Compiler transforms selected source-side representations into sparse target-native KV states. The target model can therefore perform inference using the transferred states without reprocessing the original context. We evaluate ACCT in two complementary settings comprising five general-capability benchmarks and six highly context-dependent tasks. Experiments show that ACCT matches or exceeds the target model's full-context inference performance on selected tasks, with outcomes varying across model pairs and datasets. At a context length of approximately 19K tokens, ACCT's KV state conversion achieves a 6.5-21.2x speedup over target context prefill. These results demonstrate the feasibility of transferring context computation across models under a limited state budget, offering a new approach to context reuse and efficient inference in heterogeneous model systems.