Factual Association Editing without Pretrained Knowledge Representations

We propose a closed-form factual association editing method for language models that completely eliminates the need to precompute and preserve pretrained knowledge representations. Unlike existing approaches that rely on proxy representations of pretrained knowledge obtained from sampled datasets, the proposed method formulates factual association editing as a well-conditioned optimization problem that enables stable and efficient editing using only new knowledge. During the editing process, we apply a principled treatment of ill-conditioning via within-representation centering and truncated singular value decomposition to prevent unstable or excessively amplified updates. The experimental results show that the proposed approach achieves editing performance comparable to that of state-of-the-art methods that rely on pretrained knowledge representations, while reducing amortized runtime for editing by up to 30$\times$, i.e., from 188 to 6 minutes, and requiring zero storage for pretrained knowledge representations. Even in large-scale editing (e.g., 2,000 edits), the proposed method maintains competitive editing performance, exhibiting only a 0.56% degradation on GPT-J without using any pretrained knowledge representations at all. These results provide empirical evidence that competitive factual association editing can be achieved without constructing an external proxy for the knowledge to be preserved. The code is available on an [anonymous repository](https://anonymous.4open.science/r/Anonymous-FCB2).