ResidualKV: Dynamic Knowledge Editing via an Editable Attention Key-Value Space

Knowledge editing aims to revise specific facts in large language models (LLMs) without costly full-model retraining. Existing editors largely follow a static editing paradigm, performing localized factual updates in feed-forward network (FFN) layers. While effective, this fixed editing substrate can limit factual writing and generalization across query reformulations. To overcome this limitation, we introduce Residual Key-Value Editing (ResidualKV), a dynamic knowledge-editing framework that rethinks factual writing and access through attention. ResidualKV stores corrections in a persistent editable key-value (KV) space and dynamically invokes them through attention during inference, enabling stronger factual writing and improved generalization across query reformulations. Further, to reduce interference from editable-KV writing, we propose Factual Residual Distillation (FRD), which distills only the required old-to-new factual transition into the editable KV space rather than reconstructing the complete factual state. By avoiding redundant encoding, FRD uses the limited KV capacity more efficiently, reducing disruption to pretrained model behavior while strengthening factual writing. Experiments across three recent LLM backbones and two benchmarks demonstrate strong editing efficacy and generalization, competitive post-edit fluency, and robust large-scale sequential editing. Further ablation and mechanistic analyses support the proposed design.