Locating and Editing Knowledge-Concerned Neurons in Vision Large Language Models
Abstract
Knowledge editing (KE) enables large language models (LLMs) to efficiently correct outdated or erroneous information without costly retraining. Recently, locate-then-edit has emerged as an effective paradigm due to its efficiency and interpretability, offering an alternative to memory-based approaches and methods that require training auxiliary editors. However, in vision large language models (VLLMs), factual knowledge is encoded across both visual and textual representations, making knowledge localization fundamentally different from that in text-only LLMs. Existing multimodal knowledge editing methods, such as MKE, typically rely on additional training, which may introduce out-of-distribution challenges. To address this gap, we propose **DELTA** (), to the best of our knowledge, the first fine-grained neuron-level locate-then-edit framework that is specially designed for multimodal knowledge editing and requires no auxiliary editor optimization. DELTA first identifies candidate layers by jointly modeling cross-modal fusion, feed-forward network engagement, and activation traceability. It then localizes knowledge-relevant neurons through attribution scores and visual filtering. For knowledge updating, **DELTA** constrains parameter changes toward an approximate null space to preserve unrelated knowledge, while perturbing visual embeddings to account for diverse visual evidence. Extensive experiments demonstrate that DELTA consistently outperforms existing knowledge editing approaches, highlighting the effectiveness of locate-then-edit for VLLMs and providing a step toward general-purpose multimodal knowledge editing.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.