Memorized or Memo-rized: Prior Leakage of Procedural Knowledge in Large Language Models

Agentic AI systems combine foundation models’ internal knowledge with external knowledge from context, retrieval, tools or memory modules. Conventionally, these are entangled with reasoning ability, making it hard to evaluate if a model truly generalizes under internal–external knowledge conflict. In this paper, we formalize knowledge into $\text{abstract}$ and $\text{surface-form}$ knowledge, which are components used in $\text{reasoning}$. Under this formalization, we analyze the effects of internal–external knowledge conflict and identify the $\text{prior leakage}$ phenomenon: a model’s $\text{undesirable over-reliance}$ on internal knowledge under internal–external conflict. We evaluate 17 open source models on two newly proposed procedural knowledge datasets, $\text{SQLShift}$ and $\text{PandasShift}$, where common SQL and Pandas operators are swapped or renamed. We observe substantial prior leakage across the models: the knowledge utilization stays heavily anchored on internal knowledge, even when explicitly instructed to use external knowledge. We also investigated the use of gradient ascent unlearning as a way to resolve prior leakage, which was mostly unsuccessful and did not substantially change knowledge utilization. We also attempted to control internal knowledge by post-training a vintage LLM, where training with question-answer data is observed to be vital in knowledge injection while impact of training epochs and size of training set was inconclusive. We believe this evaluation suite will support the development of trustworthy AI agents that respect user-provided external evidence over unwarranted reliance on parametric priors, paving the way toward user-sovereign AI. Code is provided in supplementary material.