acceptodds
Under review as a conference paper at ICLR 2027

Memorized or Memo-rized: Prior Leakage of Procedural Knowledge in Large Language Models

Abstract

Agentic AI systems combine foundation models’ internal knowledge with external knowledge from context, retrieval, tools or memory modules. Conventionally, these are entangled with reasoning ability, making it hard to evaluate if a model truly generalizes under internal–external knowledge conflict. In this paper, we formalize knowledge into and knowledge, which are components used in . Under this formalization, we analyze the effects of internal–external knowledge conflict and identify the phenomenon: a model’s on internal knowledge under internal–external conflict. We evaluate 17 open source models on two newly proposed procedural knowledge datasets, and , where common SQL and Pandas operators are swapped or renamed. We observe substantial prior leakage across the models: the knowledge utilization stays heavily anchored on internal knowledge, even when explicitly instructed to use external knowledge. We also investigated the use of gradient ascent unlearning as a way to resolve prior leakage, which was mostly unsuccessful and did not substantially change knowledge utilization. We also attempted to control internal knowledge by post-training a vintage LLM, where training with question-answer data is observed to be vital in knowledge injection while impact of training epochs and size of training set was inconclusive. We believe this evaluation suite will support the development of trustworthy AI agents that respect user-provided external evidence over unwarranted reliance on parametric priors, paving the way toward user-sovereign AI. Code is provided in supplementary material.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Prior Leakage, LLM agents, Large Language Models, LLM evaluation, LLM Reasoning, Agents, LLM Knowledge, Procedural Knowledge Conflict, Internal Knowledge, External Knowledge, Benchmark, Dataset

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DEgEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the behavior of large language models under procedural knowledge conflicts, namely whether models still rely on their internal priors when the SQL or Pandas operator definitions provided in context differ from those learned during pre-training. The authors distinguish between abstract knowledge and surface-form knowledge, and refer to the undesirable over-reliance on internal knowledge as "prior leakage." Two benchmark datasets are constructed by redefining or renaming code operators, and models are evaluated under several prompting conditions based on both their generated answers and code. The experiments show that external documentation and explicit instructions can encourage models to use the new knowledge, but most models still fail to fully override their internal priors.

Critical strengths
  1. The paper focuses on procedural knowledge conflict rather than the more commonly studied factual entity conflict, which gives the topic some novelty.
  2. The controlled setup, based on redefining or renaming code operators, directly tests the degree to which models follow contextual knowledge.
  3. Beyond final answers, the paper introduces metrics based on token probability distributions to measure the relative use of internal and external knowledge, providing a more fine-grained analytical perspective.
Critical weaknesses
  1. The evaluation tasks are centered on relatively simple single-step aggregation operations, so the task complexity and domain coverage are limited, making it unclear how far the conclusions generalize to more realistic and complex procedural knowledge settings.
  2. The correspondence between the proposed metrics and final model behavior is not sufficiently validated, and interpreting knowledge sources solely from token probability changes risks confounding effects.
  3. The unlearning and controlled training experiments are weak and admit multiple interpretations, so they do not strongly support the paper's claims about the causes or mitigation of prior leakage.
Questions for authors
  1. Can the authors clarify whether the observed conclusions hold in more complex multi-step program synthesis, other programming languages, or other procedural knowledge domains?
  2. How do the authors rule out the possibility that token probability changes reflect linguistic preferences, formatting constraints, or positional effects rather than genuine differences between internal and external knowledge use?
  3. Given that the unlearning results admit multiple interpretations, how should successful forgetting be defined, and how can its causal relationship with changes in prior leakage be established more rigorously?
Reviewer_sYGfOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies procedural prior leakage: whether LLMs follow externally supplied rules that conflict with familiar coding conventions. The authors introduce SQLShift and PandasShift, evaluate many models using output accuracy and two probability-based metrics, and find persistent reliance on conventional operators despite conflicting documentation and explicit instructions.

Critical strengths

Adapting to context-specific rules is a practical requirement for LLMs, particularly when working with updated APIs, custom libraries, and user-defined interfaces. The paper addresses this relevant problem.

Critical weaknesses
  1. The aggressive semantic reassignment of established symbols such as SELECT and MIN is far removed from typical API updates or custom interfaces. The relevance of these findings to realistic procedural knowledge conflicts remains unclear.

  2. Interference from familiar coding conventions is an expected outcome. Although the paper explores unlearning and controlled training, these experiments offer limited insight into the causes of this interference or how to predict and mitigate it.

Questions for authors

What evidence supports the relevance of the observed leakage to realistic API updates or custom interfaces, beyond the aggressive operator remappings used here?

Reviewer_zyMuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies “prior leakage”: models’ reliance on familiar procedural knowledge despite conflicting externally supplied documentation. It introduces SQLSHIFT and PANDASSHIFT, which change operator meanings or names while preserving the underlying computational task, alongside two probability-based metrics, Knowledge Anchor and Knowledge Shift. Experiments across 17 models reveal incomplete adaptation to the supplied definitions. Additional unlearning and vintage-model training experiments explore potential explanations, although their results remain inconclusive.

Critical strengths
  1. Relevant question and controlled design. The paper addresses an important limitation of systems that must follow updated documentation rather than memorized conventions. Holding the underlying task fixed while changing operator mappings provides a useful setting for studying this problem.

  2. Broad model coverage and informative comparisons. The evaluation includes 17 models spanning 4B–405B parameters, general-purpose, coding, and thinking variants, four prompting conditions, and different amounts of documentation. These comparisons show that adaptation varies substantially and is not reliably solved by model scale or explicit instructions.

  3. Analysis beyond final-answer accuracy. Combining code execution, string matching, partial operator adoption, and token-probability diagnostics provides a more informative picture of model behavior than a single aggregate accuracy score.

Critical weaknesses
  1. The benchmark’s task coverage is narrow. Both datasets use 104 questions over ten generated tables, involving selection and a single aggregation operator. This simplicity is useful for controlled diagnosis, but limits conclusions about broader procedural reasoning or agentic adaptation. Moreover, familiar-operator swaps and novel-name replacements are averaged together; reporting them separately would help distinguish interference from familiar associations from difficulty learning unfamiliar mappings.

  2. The interpretation and implementation of the probability metrics need clarification. Knowledge Anchor and Knowledge Shift measure preferences and changes in operator probabilities, rather than directly identifying the source of knowledge used. Claims about “exact usage” therefore seem stronger than the evidence supports. The paper should validate these metrics against the reported behavioral outcomes and clarify how multi-token operators, renamed strings, and differing generated prefixes are handled. Matching character length does not necessarily control tokenization.

  3. The intervention experiments provide limited explanatory evidence. Unlearning generally fails to remove the relevant SQL capability, while the vintage model requires task-format fine-tuning and never successfully adopts the counterfactual operators. Consequently, these experiments cannot cleanly distinguish persistent priors from instruction-following or remapping limitations. The authors appropriately acknowledge the inconclusive results, but stronger controls would be needed for mechanistic conclusions.

Questions for authors

See weakness

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

Hi reviewers, we have uploaded the revised PDF based on questions discussed here. The revisions adds the following (see new text in red in the pdf):

  • [New data] SQLShift with increased task complexity (Appendix A; Reviewer DEgE): Finding: leakage grows for complicated multi-step queries (mean KA −0.33 → −0.80).
  • [New experiment] KA/KS robustness under paraphrasing (Appendix C; Reviewer DEgE): formal and casual prompts. Finding: KA/KS stay stable across tone paraphrasing. Casual tone might cause the model to leak more.
  • [New data] Three datasets simulating real-world programming changes, such as API updates and user-customized interfaces (Appendix B; Reviewers zyMu and sYGf): KeywordShift, EditShift and DialectShift. Finding: Leakage appears where a familiar symbol has another meaning.
  • [New analysis] Reasoning trace analysis and operator embedding similarity (Appendix D; Reviewer sYGf). Finding: Naming similarity does not explain which names leak. The model usually attribute the external customized documentation as a typo/mistake and keeps its prior, rather than tuning itself to the new interface.

You can find details in our replies to each reviewer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.