acceptodds
Under review as a conference paper at ICLR 2027

Understanding Contextual Recall in Transformers: How Finetuning Enables In-Context Reasoning over Pretraining Knowledge

Abstract

Transformer-based language models excel at in-context learning (ICL), where they can adapt to new tasks based on contextual examples, without parameter updates. In a specific form of ICL, which we refer to as contextual recall, models pretrained on open-ended text leverage pairwise examples to recall specific facts in novel prompt formats. We investigate whether contextual recall emerges from pretraining alone, what finetuning is required, and what mechanisms drive the necessary representations. We introduce a controlled synthetic framework where pretraining sequences consist of subject-grammar-attribute tuples, with attribute types tied to grammar statistics. We demonstrate that while such pretraining successfully yields factual knowledge, it is insufficient for contextual recall: models fail to implicitly infer attribute types when the grammar statistics are removed in ICL prompts. However, we show that finetuning on tasks that require implicit inference, but are distinct from the ICL evaluation, using a subset of subjects, triggers the emergence of contextual recall. This transition is accompanied by the formation of low-dimensional latent encodings of the shared attribute type. For mechanistic insight, we derive a construction for an attention-only model to succeed on factual and contextual recall, corroborated by empirical validation, and analysis of the training dynamics for the transition from factual to contextual recall.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
contextual recall, transformers, in-context learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Sr7wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies contextual recall: inferring the queried attribute type from in-context examples, then retrieving the query subject's stored value of that type. A two-layer transformer pretrained on synthetic single-subject sequences (PT), where type-specific grammar precedes each attribute, learns the facts but fails on in-context prompts (Finding 1). Finetuning on half of the subjects, with plain in-context sequences (ICL) or with one to five grammar tokens before each attribute (ICL+Grm), generalizes to the other half (Findings 2-3); after ICL+Grm finetuning, layer-2 representations cluster by type (Finding 4). In a one-layer attention-only model with explicit relation tokens, one or two population gradient steps turn a factual-recall construction into a contextual-recall one.

Critical strengths
  1. The question is well posed, and the held-out-subject test probes it cleanly. Retrieving a stored fact through a relation the context only implies ties in-weights knowledge to in-context inference, and held-out subjects separate learning an access mechanism from memorizing new pairs. The synthetic generator also makes the controls I request below cheap to run.

  2. The constructions are checked against trained models. In the reported runs, the trained one-layer heads match the constructed subject and relation heads, so the constructions describe what training found.

Critical weaknesses
  1. Format and the pretraining data confound the explanations of Findings 1 and 3. Pretraining places each attribute after a long grammar sequence, whereas in-context prompts ask for it right after the subject. The paper attributes Finding 1 to reliance on grammar statistics (p.4 l.174-175). Yet on ICL+Grm sequences that restore the type-specific grammar at its pretraining length, the model at iteration 0 almost always outputs an attribute but picks the right type no better than chance (Fig. 10, left; Q1). These sequences are over three times longer than PT sequences, which confounds the test. Finding 3's claim that short ICL+Grm sequences "serve the same purpose as ICL sequences" (p.5 l.224) also assumes, without quantifying it, that one to five grammar tokens cannot reliably encode the type (Q3). On the data side, no pretraining sequence contains same-type attributes of different subjects, whereas base LLMs solve knowledge ICL tasks such as country-capital (Hendel et al., 2023; Todd et al., 2024). The paper cites Allen-Zhu & Li (Part 3.1) for the same held-out-subject result but leaves out their condition that extraction depends on augmented pretraining data. This generator's fresh grammar and randomized attribute order resemble such augmentation, which may be why finetuning succeeds here. The title credits finetuning with enabling in-context reasoning, but the evidence does not yet separate missing type inference from a format mismatch, or a need for finetuning from a gap in this pretraining data. I would ask for three things: (i) evaluating the pretrained model on PT-length ICL+Grm sequences with type-specific versus type-uninformative grammar; (ii) finetuning on ICL+Grm with an independent type per demonstration, so only the format can be learned; and (iii) discussing the Allen-Zhu & Li condition, then either narrowing the title or showing that adding a small fraction of multi-subject, same-type sequences to pretraining does not yield contextual recall without finetuning.

  2. The Sec. 4 theory covers a different setting from Sec. 3, and its gradient-step proof contains errors as written. By design (p.6 l.317), the theory models only Findings 1-2, not the ICL+Grm recipe of Finding 3. Prop. 2 also requires every attribute value to appear during finetuning. Sec. 3, however, has as many type-1 values as subjects; if they map one-to-one, the held-out subjects' type-1 values never appear in finetuning, yet held-out accuracy is near perfect (Fig. 3; Q2). The two-layer model may therefore use a route outside the construction. Finally, Lemma C.1 bounds every output probability below one over the vocabulary size, although the probabilities must sum to one, and the step at p.25 l.1332 treats their sum as much smaller than one. Because the theory is the paper's mechanistic explanation, these gaps leave the mechanism behind the main recipe unestablished. The lemma and that step look repairable, but until they are corrected, Theorem 1 is unverified. (I checked only this lemma and step.)

Questions for authors
  1. Is iteration 0 in Figs. 3 and 10 the pretrained checkpoint? Restricted to attribute tokens, does it pick the right type above chance?

  2. How are attribute values assigned to subjects, and what is held-out accuracy per type?

  3. What Bayes-optimal type accuracy do the Finding 3 grammar tokens allow, pooled over all subsequences in the context?

  4. Why does held-out accuracy under the Finding 3 recipe fall to about a quarter with eight layers and to near zero with four heads (Fig. 12)? Figs. 3, 4, and 12 also appear to be single runs; do the main results hold across at least five seeds, with confidence intervals?

  5. With disjoint attribute vocabularies, one demonstration identifies the queried type, and held-out accuracy saturates after the first (Fig. 6, left). Does the model aggregate information across demonstrations when one is not enough? Sharing a controlled fraction of attribute tokens across types would test this; until then, I would narrow the claim that the model "aggregates information from multiple examples" (p.6 l.276-277).

3 comments

Thank you for your careful reading and detailed feedback. We are encouraged that you find our research question well-motivated and the held-out-subject evaluation useful, and appreciate the empirical validation of our theoretical constructions. We respond to your concerns below.

W1(i), Q1: Performance of the pretrained model on ICL+Grm sequences

Thank you for the question. We verified that the ICL+Grm sequences in Fig. 10 have the same length as the PT sequences (not 3 times longer), we have clarified this in the App. B in the updated version. As you pointed out, in this case, at the start of FT, the model always outputs an attribute token, whereas it takes a few iterations to get attribute type and exact match correct.

Our statement in lines 174-176 refers specifically to PT sequences, where the model learns to use explicit grammar statistics to infer the attribute type and retrieve the corresponding fact. This does not necessarily imply that the same ability transfers directly to ICL+Grm sequences. Although the grammar length and subsequence structure are the same, the overall data distribution differs: PT sequences contain a single subject, whereas ICL+Grm sequences contain multiple subjects. Nevertheless, the two settings share the underlying grammar-based attribute retrieval task, and Fig. 10 (left) shows that the model requires only a few fine-tuning iterations to achieve high attribute-type and exact-match accuracy on ICL+Grm sequences. We have revised lines 174-176 in the updated version to mention that the pretrained model fails to generalize to grammar-free ICL sequences, but not attribute this failure specifically to its reliance on grammar statistics.

W1(ii), Q3: Informativeness of short grammar sequences

Thank you for the suggestion about better isolating the role of FT on sequences that encourage adaptation to format change vs implicit inference. We conducted the proposed experiment, where we finetune on ICL+Grm sequences with short grammar length, but with attribute types sampled independently rather than shared across demonstrations. We find that the model learns to adapt to the new sequence format and predict attribute tokens, but fails to implicitly infer the relevant attribute type from the in-context examples. Consequently, ICL accuracy improves only slightly. This highlights the importance of shared attribute types during finetuning for learning the implicit inference mechanism. We have included these results in Fig. 11 of the updated paper.

Next, regarding your question about the Bayes optimal accuracy for ICL+Gram sequences, We would like to highlight the results in Fig. 9 (left), where we evaluate the finetuned model on ICL+Grm sequences with Sft=1S_{ft}=1. In this case, each grammar token is sampled uniformly, independently of the attribute type. Thus, even when pooling information across all subsequences in the context, the Bayes-optimal attribute-type accuracy using grammar tokens alone is exactly 1/L=12.5%1/L=12.5\%. Nevertheless, the finetuned model achieves high accuracy in this setting, demonstrating that it can infer the attribute type from the in-context attribute tokens rather than relying solely on grammar statistics.

More generally, during finetuning, we sample Sft∼Unif({1,…,5})S_{\mathrm{ft}}\sim\mathrm{Unif}(\{1,\ldots,5\}). Even assuming perfect grammar-based type inference for Sft≥2S_{ft} \geq 2, the average Bayes-optimal grammar-only accuracy is upper-bounded by 15⋅18+45⋅1=82.5\frac15\cdot\frac18+\frac45\cdot1=82.5%.

This is a conservative bound, whereas the finetuned model achieves near perfect accuracy, further indicating that it uses information beyond grammar statistics alone. We will clarify this in the updated version.

W1(iii): Role of PT augmentation and necessity of finetuning

Thank you for the question. Regarding your comment about the role of PT augmentation in Allen-Zhu and Li (2023), we have clarified this aspect of their findings in App. A in the updated version.

Regarding the suggested experiment, we agree that alternative PT distributions, such as mixtures containing multi-subject, same-type sequences, with shorter grammar lengths, could potentially induce contextual recall without a separate FT stage. Our main goal is to understand how factual knowledge acquired during PT can be leveraged for contextual recall after FT, and the role of different types of data, and we use stage-wise PT and FT to study this in a controlled setting. Thus, we do not claim that FT is universally necessary, but rather that the PT distribution with same subject sequences is insufficient, while FT on sequences requiring implicit attribute-type inference enables contextual recall. We will clarify the scope of this claim in the updated version.

W2: Regarding the theoretical results in Sec. 4

We thank the reviewer for the careful examination of our theoretical analysis. We address the concerns below.

Revised Lemma C.1 and related steps. Thank you for pointing out the error in the probability bound in Lemma C.1 and related approximation steps in the proof of Theorem 2. We have revised App. C.2 to address these issues. Specifically, Lemma C.1 now establishes πv≤1/(V−LUL)\pi_v\leq 1/(V-LU_L), which implies ∑vπv2≤max⁡vπv≤1/(V−LUL)\sum_v\pi_v^2\leq \max_v\pi_v\leq 1/(V-LU_L). We use this bound to control the softmax contribution in the relation-head gradient approximation, and have revised the subsequent gradient approximations to ensure consistency throughout the proof.

Scope of the theoretical results. As noted in Sec. 4, our theoretical analysis considers a simpler setting to understand the role of PT and FT in enabling contextual recall, rather than providing a guarantee for the ICL+Grm finetuning procedure in Finding 3. Our analysis shows that under certain conditions, a few steps of GD starting from the PT construction can enable contextual recall. This allows us to concretely see how the subject-head mechanism learned during PT is retained, while the relation-head mechanism is repurposed during FT to infer the relevant attribute type from the in-context examples. We present this as a candidate mechanism, without claiming that it is the only mechanism through which the model succeeds on contextual recall.

Q2: Assignment of attribute values and generalization across attribute types

Thank you for the question. In our main experiments, we use M=256M=256 subjects and L=8L=8 attribute types, with M1=256M_1=256 unique attribute values for type 1 and M2=⋯=M8=32M_2=\cdots=M_8=32 for the remaining types. For type 1, each subject is assigned a unique attribute value. We have included results for ICL accuracy evaluated separately for each attribute type, for both seen and held-out query subjects (Fig. 12 in the updated version). We find that the model achieves nearly perfect accuracy on held-out subjects even for type 1, where the corresponding attribute values are never observed during FT.

Q4: Effect of model scale and robustness across initialization seeds

Thank you for these questions. As shown in Fig. 12 (Fig. 14 in the updated version), while all architectures achieve good factual recall after PT, their generalization to ICL sequences after FT is not monotonic with model scale. While we do not have a definitive explanation for this behavior, one possibility is that different models may learn different representations of factual knowledge during PT, some of which are more amenable to generalization after FT. Indeed, Fig. 24 (Fig. 26 in the updated version) shows that models achieving perfect factual recall can learn different representations, leading to different generalization performance after FT. While this does not directly explain the model scale behaviour, it suggests that the representations learned during PT may play an important role beyond factual recall accuracy alone.

Thank you for the suggestion on including results across multiple initialization seeds, for the main experiments. We will include those in the updated version of the paper.

Q5: Role of multiple demonstrations in attribute-type inference

Good question. We agree that in our setting, where attribute vocabularies are disjoint, a single demonstration is sufficient to identify the attribute type. Our main observation in Fig. 6 is that increasing the number of demonstrations leads to stronger clustering of the learned attribute-type representations, even when the accuracy has already saturated. Thus, our results characterize how the representation geometry evolves with the number of examples, rather than establishing that multiple demonstrations are necessary for identifying the attribute type. We agree that studying settings with overlapping attribute vocabularies, where multiple examples are needed for disambiguation, would be an interesting extension.

We hope these clarifications address your concerns and that you would consider increasing your score.

Thank you for your responses. The revised probability bound is a sensible correction. After carefully considering the paper and your responses, I have decided to maintain my score.

Reviewer_GZoYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how a model can use examples in a prompt to figure out which facts to retrieve from its pretrained knowledge (contextual recall). The authors use a synthetic dataset to show that pretraining helps with factual knowledge but lacks in contextual recall, unless they receive further training. They showed that further training helps the model identify patterns in examples and use those patterns to retrieve the right information even on new subjects that they were not fine tuned on.

Critical strengths
  • synthetic set up makes it easy to see the impact of pretraining and finetuning in contextual recall
  • the authors compare different types of fine tuning and show that the way training examples are structured can make a big difference
  • authors support their findings with both experiments and mathematical explanations of how the model might be working
Critical weaknesses
  • paper mostly focuses on retrieving simple facts; unsure whether the findings hold for more complicated reasoning tasks
  • it seems like most new subjects introduced share possible attribute values with subjects used during fine tuning, so I wonder how they would perform with completely new info
  • some larger models actually perform worse in the experiments but the paper doesn’t fully explain why (Appendix B.2, Fig 12)
Questions for authors
  • do you think your results would hold for real LMs trained on natural text rather than synthetic examples?
  • why do some larger models perform worse than smaller ones?
  • could you test whether internal patterns the models learn are actually necessary for it to retrieve the correct answers? Rather than them just appearing alongside good performance?
1 comment

Thank you for your careful reading and positive assessment of our work. We are encouraged that you find our controlled setup and comparison of different finetuning distributions informative, and appreciate the supporting mechanistic analysis. We respond to your questions below.

Q1: Do we expect our findings to extend to LMs trained on natural text?

Thank you for the question. We believe that the main insights from our work are relevant to LMs trained on natural text as well. Recent work studying ICL mechanisms in pretrained LLMs (Hendel et al., 2023; Todd et al., 2024; Yin & Steinhardt, 2025) presents evidence for the formation of task vectors that encode the input–output relationship of the ICL task. Our work provides insights into how these representations emerge during training.

One important difference is that LMs are pretrained on much more diverse data, which may already contain sequences that encourage implicit inference of the relevant attribute type, allowing contextual recall to emerge without a separate FT stage. This is consistent with our findings, which highlight the importance of training on sequences that require implicit inference, rather than simply acquiring factual knowledge. Our controlled setup allows us to isolate the role of different types of training data and provide a mechanistic explanation for how this capability emerges, which is challenging to study directly in LMs pretrained on internet-scale data.

W3, Q2: Why do some larger models perform worse in Fig. 12?

In Fig. 12 (Fig. 14 in the updated version), we vary the model depth, number of heads, and embedding dimension while keeping the training settings fixed. We find that all these models achieve good factual recall after PT, but their generalization to ICL sequences after FT is not monotonic with model scale.

While we do not have a definitive explanation for this behavior, one possibility is that different models may learn different representations of factual knowledge during PT, some of which are more amenable to generalization after FT. Indeed, in a different setting, Fig. 24 (Fig. 26 in the updated version) shows that models achieving perfect factual recall can learn representations that differ in how closely they match our theoretical construction, leading to different generalization performance after FT. While this does not directly explain the model scale behaviour, it suggests that how factual knowledge is represented during PT may play an important role in downstream generalization. More broadly, investigating how different model architectures affect the representations learned during pretraining, and consequently their downstream generalization after finetuning, is an interesting direction for future work.

Q3: Are the learned attribute-type representations necessary for contextual recall?

Thank you for the suggestion. Our current results in Fig. 6 show that the learned attribute-type representations can causally influence model predictions. To investigate whether these representations are necessary for contextual recall, as you suggested, we will perform an additional intervention where we remove the attribute-type-specific component from the layer-2 attention representations and evaluate the resulting ICL performance on held-out subjects. We will also compare against removing a random subspace of the same dimension to control for the effect of perturbing the representations. We will include these results in the updated version of the paper.

Reviewer_qUyVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies contextual recall in a controlled synthetic setting: inferring the relevant attribute type from in-context examples and retrieving the corresponding fact learned during pretraining. The authors show that their pretraining setup supports factual recall but does not, by itself, yield contextual recall. Finetuning on a subset of subjects enables generalization to held-out query subjects whose facts were seen during pretraining, including transfer from finetuning sequences containing short grammar spans to evaluation sequences without grammar. The paper combines representation analysis, activation interventions, and theoretical analysis of a simplified attention-only model to investigate how this capability may emerge.

Critical strengths
  1. The experiments are designed around the central explanatory question. The paper goes beyond reporting improved performance after finetuning. Held-out-subject evaluation and cross-format transfer, together with the contrast between short-grammar transfer and poor transfer with long grammar, help distinguish learning a transferable access mechanism from merely memorizing answers for finetuning subjects. This makes the empirical account of how the capability emerges more informative.
  2. The mechanistic analysis combines several forms of evidence. The authors examine internal representations and intervene on them to steer outputs, then provide explicit constructions, an analysis of finetuning dynamics, and comparisons with trained models in a simplified setting. These complementary analyses offer a concrete account of how task information may be combined with stored factual associations, rather than relying on behavioral results alone.
Critical weaknesses

My main reservation concerns the extent of the new mechanistic insight. Prior work has examined both the distinction between knowledge storage and extraction and the role of task vectors in representing contextual task information. The paper provides a useful account of how task inference and factual retrieval interact under specific synthetic data distributions. Its transfer experiments and mechanistic analysis add detail about when and how this connection can be learned. However, I find that the mechanism itself offers a relatively modest conceptual advance over existing accounts of knowledge extraction and task vectors.

Questions for authors

Appendix A notes that different initializations can lead to a subject-head mechanism closer to that described in prior work. If you have examined models with this alternative mechanism, do they exhibit similar generalization to held-out query subjects after finetuning? Any existing observations would help clarify whether the proposed account of contextual recall also applies across these different subject-head implementations.

1 comment

Thank you for your careful reading and positive assessment of our work. We are encouraged that you find our transfer experiments informative and appreciate the complementary empirical and theoretical analyses supporting our mechanistic account of contextual recall.

Q1: Generalization across different subject-head mechanisms.

Thank you for the question. We have observed that models with both types of subject-head mechanisms generalize to ICL sequences with held-out query subjects after finetuning. As discussed in App. A, while our construction uses a subject head that suppresses attributes not associated with the query subject, different initialization seeds can lead to a mechanism similar to Nichani et al. (2024), where the subject head instead boosts attributes associated with the query subject. Importantly, both mechanisms serve the same functional role of retrieving subject-specific factual information learned during pretraining, which is combined with the attribute-type information inferred from in-context examples. Contextual recall additionally requires inferring the relevant attribute type from the in-context examples, which is facilitated by the relation head after finetuning. This suggests that the overall mechanism for contextual recall does not depend on the specific implementation of the subject head.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.