acceptodds
Under review as a conference paper at ICLR 2027

Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning

Abstract

Fine-tuning LLMs to inject new knowledge faces a critical challenge: LLMs can quickly memorize new facts, yet fail to use them for downstream reasoning tasks. We formalize this failure as the Knowing–Using Gap, characterized by an accuracy gap and a temporal lag between memorization and generalization. To understand this phenomenon, we fine-tune LLMs with unseen knowledge and monitor the spatial permeation dynamics of the knowledge internally with self-patching, an adaptation of activation patching that scans all layer pairs at every fine-tuning checkpoint. Self-patching identifies activation locations where relocating representations substantially improves failed generalization cases. These results are consistent with a knowledge-circuit misalignment hypothesis: memorized representations can exist internally but may not be routed to computation-effective layers. Experiments are done cross-domain for the robustness of this finding. Building on this diagnosis, we propose layer-wise representation self-distillation (LRSD) that aligns knowledge representation from late storage layer to middle layer. LRSD keeps improving generalization after fine-tuning saturates and nearly doubles multi-hop chaining accuracy on Qwen, while leaving memorization intact. Code and data are open at https://anonymous.4open.science/r/Mem2Gen-71FF.

Primary area:
interpretability and explainable AI
Keywords:
mechnistic interpretabiltiy, fine-tuning dynamics, continual learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_igKBOpenReview ↗
Rating
1
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the Knowing–Using Gap, in which fine-tuned models can memorize new facts but fail to use them for multi-hop reasoning. It characterizes this phenomenon through an accuracy gap and a temporal lag between memorization and generalization. The authors propose self-patching, which copies hidden states at entity positions from one layer to another within the same model, scans for layer pairs that recover correct answers, and tracks how these effects evolve during training. Across two knowledge domains and six models, selecting the optimal layer pair for each sample substantially improves reasoning accuracy. Based on these results, the authors propose a knowledge–circuit misalignment hypothesis and introduce LRSD to align middle-layer entity representations with deeper-layer representations. LRSD improves chaining accuracy across four models without test-time intervention while largely preserving factual recall.

Critical strengths
  1. The paper provides systematic interventional evidence for the gap between memorization and knowledge use. Self-patching does not require activations from a separate correct run. Instead, it reuses representations from the failed query itself. Table 2 shows substantial recoverable headroom across models and both domains. For example, Qwen2.5-3B’s chaining accuracy on STaRK-Prime increases from 11.4% to 54.2%. The unrelated-fact controls and cross-context patching experiments further support the interpretation that the gains involve entity- and fact-related information. This is the paper’s most valuable contribution.

  2. The mechanistic analysis leads to a training method that requires no test-time intervention, with improvements supported by multiple runs. LRSD trains only on single-hop facts and uses no multi-hop answer supervision. In Table 9, all seed-paired comparisons across four models and three seeds show improvements, with Qwen2.5-3B’s chaining accuracy increasing from 9.6% to 18.7%. Although absolute accuracy remains low, this result extends the contribution beyond per-sample oracle intervention and strengthens the evidence for the proposed explanation.

Critical weaknesses

The code repository linked in the submission contains author names and institutional affiliations, directly revealing the authors’ identities. This violates the ICLR 2027 double-blind submission requirements, which require anonymous code repositories and specify desk rejection for submissions that disclose author identities in the paper or supplementary materials. I recommend desk rejection on this policy ground, subject to verification by the Area Chair and Program Chairs.

Questions for authors

See weaknesses.

Reviewer_xcNDOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper studies the effect of inserting new facts into languages models on downstream recall capabilities. Specifically, they show that there is a "knowing-using" gap, where a model after fine-tuned on new knowledge can recall it, but cannot use it in more complicated multi-hop tasks. The authors attribute this gap to suboptimal routing (i.e. the model has successfully stored the knowledge relation, but cannot route it to the multi-hop or intersection modules of the network).

Critical strengths

Conceptually the work is interesting, I think the dichotomy between injecting information into a model by way of memorization versus the model actually being able to use the information on more complex downstream tasks is an under-explored area. In this way, I appreciate the framing of the work.

Critical weaknesses

Throughout, the authors assume the model has circuits that can already solve multi-hop or intersection reasoning. There seems to be a missing baseline:

  1. Take relations and facts that the base model already has stored.
  2. Check to see if the model without any sort of intervention is able to solve the chaining and intersection task.

If the model is not able to do this, then the knowing-using gap is not a function of the badness of the fine-tuning procedure, but a function of the base model simply being weak. This to me seems plausible given that patching yields such a drastic increase in performance whereas explicitly building in this patching behavior only marginally improves the model.

I also believe that there are major errors in the formalization of Section 2.2. For example, TgenT_{\text{gen}} is only finite if the model achieves maximal generality at some point. How exactly is this being measured in Figure 1 (left), because the model presumably never reaches maximal accuracy on chaining? Along these lines, the temporal lag defined using TgenT_{\text{gen}} also seems dubious to me.

Questions for authors
  • I'm very confused by what Figure 4 is showing. What does the color represent here?
  • Also, (still looking at Figure 4) looking at the series of 7 heat plots on the bottom row, in the "generalized regime" why would almost all of this heat map be red? This would mean that self-patching the fact still significantly improves performance even after we have generalized and chaining accuracy is maximal, how is this possible?
  • How does finetuning with LRSD affect base model performance? My understanding is that Equation 3 can also be viewed as we are encouraging the model to create a fixed point. In this sense, the optimal solution to this objective would be to simply create attention sinks (or equivalent no-ops) in all layers between ltgtl_{\text{tgt}} and lsrcl_{\text{src}}?
Reviewer_J44wOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why fine-tuned LLMs can memorize newly injected facts yet fail to apply them in downstream reasoning, formalizing this phenomenon as the Knowing–Using Gap in accuracy and learning time. The authors introduce self-patching, which relocates entity representations across layers and tracks intervention effects throughout fine-tuning. Experiments across two knowledge domains and six models provide evidence consistent with a knowledge–circuit misalignment hypothesis. Building on this diagnosis, layer-wise representation self-distillation (LRSD) aligns late-layer representations with middle-layer representations during training. LRSD improves chaining across four models without test-time patching, nearly doubling accuracy on the two Qwen models.

Critical strengths
  1. The paper addresses an important question and goes beyond documenting a behavioral failure. Connecting memorization dynamics, representation location, and downstream usability offers an informative perspective on knowledge injection and compositional generalization.

  2. The mechanistic investigation is systematic and supported by complementary controls. Longitudinal layer-pair scans, cross-domain replication, token-position ablations, irrelevant-state patching, and cross-context consistency collectively strengthen the interpretation. The figures clearly communicate the progression from memorization to intervention-enabled use.

  3. LRSD provides a compelling link between diagnosis and intervention. It improves generalization using only single-hop supervision, with no test-time oracle or additional forward pass. Three-seed comparisons, paired statistical testing, and layer/position controls make this proof of concept substantially more convincing than an interpretability analysis alone.

Critical weaknesses
  1. Self-patching demonstrates that relocating representations can restore correct answers, but does not uniquely establish misrouting as the cause of failure. Whole residual-stream replacement changes multiple features, and the strongest recovery results select the best layer pair per question. The controls are valuable, but finer-grained necessity tests or held-out fixed-pair interventions would strengthen the distinction between knowledge storage and reasoning computation. The categorical conclusion that this is “a routing problem, not a capacity one” should be qualified.

  2. Temporal lag excludes runs that never generalize, so it describes successful cases rather than the complete training population. Appendix B also defines memorization time by first successful recall, while generalization requires ten consecutive correct epochs. Reporting matched stability criteria, the fraction of excluded runs, and sensitivity to the stability window would make comparisons across training schemes easier to interpret.

  3. LRSD is evaluated on STaRK-Prime chaining with 1B–3B models, whereas the diagnostic analysis covers broader domains and scales. Testing LRSD on STaRK-MAG, intersection tasks, and larger models would establish how widely the proposed remedy transfers. Its absolute chaining accuracy also remains below 20%, and general-capability preservation is untested, so the practical claims should remain appropriately scoped.

Questions for authors

Does the layer pair selected on Qwen2.5-3B transfer to independently sampled facts and unseen relation compositions without further selection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.