acceptodds
Under review as a conference paper at ICLR 2027

Knowledge That Is Linear Before Fine-Tuning Generalizes After It

Abstract

Not all facts a language model can recall support further learning equally well. We distinguish active facts, which support generalization beyond their own recall, from inert facts, which can be recalled but do not support such generalization in a given task. We study this distinction by constructing new relations from existing ones. Take the relation that maps a place to its continent (Paris → Europe) and assign each of its answers an arbitrary new label (Europe → crab, Asia → zebra). The new relation then maps Paris → crab. We fine-tune the model on the new relation for a subset of subjects, without telling it the original relation or the relabeling rule, and test whether it assigns the corresponding labels to held-out subjects that share an answer in the original relation. Having learned Paris → crab, does it answer Munich → crab? Across eleven language models and 70 relations, how linearly readable the original relation is in pretrained representations strongly predicts this generalization. Within each model, the two correlate across relations at Pearson r = 0.87 to 0.92. This association persists when controlling, separately and jointly, for the number of fine-tuning subjects and answers, the model's accuracy on the original relation, and how often the subjects occur in pretraining text. It is also robust to the arbitrary choices in the construction. Re-randomizing the label assignment and fine-tuning seed, or replacing the animal labels with labels from thirteen other semantic categories, leaves it essentially unchanged. When a fine-tuned model errs, as in Munich → zebra, “zebra” stands for Asia, and Asia is more often among the linear probe's top answers for Munich than among the pretrained model's own top answers to the original question. These findings tie the active/inert distinction to linear readability. Whether a new relation generalizes depends less on whether its facts can be recalled than on how readily their shared structure can be read out linearly.

Primary area:
interpretability and explainable AI
Keywords:
llm, language models, interpretability, probing, linearity, generalization, relations, fine-tuning, linear probes, linear representations, factual knowledge, relational knowledge

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NUj4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether the linear readability of pretrained relational knowledge predicts generalization after fine-tuning. The authors first measure the linear readability of selected relations, then synthesize new relations by randomly relabeling their answers, fine-tune the models on a subset of subjects, and test them on held-out subjects. On 11 models, they show that pre-linearity strongly correlates with the generalization performance they define.

Critical strengths
  • The authors validate their findings across 11 models and 70 relations drawn from multiple data sources.
  • The paper describes its methodology and experimental settings in detail, supporting the reproducibility of the results.
Critical weaknesses
  • I have concerns about the paper's contribution. All experiments are conducted in a toy, synthetic setting. The analysis is largely descriptive and provides limited insight into the underlying reasons, such as why pre-linearity would associated with generalization. The paper also offers little practical guidance on how this property could be applied in actual model training.
  • I also have concerns about the rigor of the experimental conclusions, the interpretation of the observed results, and what the authors regard as generalization. I elaborate on these concerns in the Questions section.
Questions for authors
  • Regarding the experimental setup and interpretation:
    • Using the authors' Italy and Munich example, how can we establish that the model answers Munich → crab by learning the Europe → crab mapping or by using the fact that both Italy and Munich are in Europe? Could there be alternative explanations? For example, the model might simply rely on similarity between subjects: if Italy is the training subject most similar to Munich, it might assign Munich the same label, crab. Would other subjects associated with Italy, such as pizza or Galileo, also be mapped to crab?
    • My previous point concerns both how generalization is defined in this paper and how it is inferred from the experiments. If the authors attribute the Munich → crab prediction to the model's knowledge of the location-to-continent relation, could this generalization be tested more directly? For example, could they test whether the model has learned the equivalence between Europe and crab?
    • Generalization should also be distinguished from overgeneralization. If the model has indeed learned that only locations in Europe should be assigned crab and only locations in Asia should be assigned zebra, would it avoid predicting crab or zebra for a city on a continent not covered by the experiments?
  • Lines 254–255: Why is the query template needed when probing the representation of Paris? In a causal language model, shouldn't the representation at Paris be unaffected by the subsequent text, such as "is located in"?
  • The Pre-linearity paragraph in Section 3 would be easier to follow with pseudocode outlining the procedure.
2 comments

We thank the reviewer for recognizing that the findings are validated across 11 models and 70 relations drawn from multiple data sources, and that the detailed description of the methodology supports reproducibility.

We respond to the main concern, that the experiments use a toy, synthetic setting.

Only the new answers are synthetic. The models are pretrained language models, and the relations are real relations they learned in pretraining. We invent the new answers because the question requires it. To measure how a model generalizes knowledge it has just been taught, that knowledge must be certain to be absent from pretraining, and its tie to existing knowledge must be controlled. Real facts guarantee neither.

This design is established practice at ICLR and other top venues. Berglund et al. [1] establish the reversal curse by fine-tuning language models on invented facts about fictitious people. Feng et al. [2] fine-tune on facts about fictitious people. Huang et al. [3] follow this design. Jain et al. [4] study the effects of fine-tuning in synthetic, controlled settings.

[1] Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, Owain Evans. The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A". ICLR 2024.

[2] Jiahai Feng, Stuart Russell, Jacob Steinhardt. Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts. ICML 2025.

[3] Yixiao Huang, Hanlin Zhu, Tianyu Guo, Jiantao Jiao, Somayeh Sojoudi, Michael I. Jordan, Stuart Russell, Song Mei. Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformers. NeurIPS 2025.

[4] Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, et al. Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks. ICLR 2024.

Most of my concerns have not been addressed. I maintain my original rating.

Reviewer_yF7hOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why some facts encoded in a language model support further learning and generalization, while others can be recalled but offer limited support for transfer. The authors construct new relations by randomly relabeling the answers of existing factual relations, fine-tune models on a subset of subjects, and evaluate the new labels on held-out subjects. Across eleven models and seventy relations, the cross-validated linear readability of pretrained subject representations strongly predicts post-fine-tuning generalization, with Pearson correlations of 0.87–0.92. This association persists after controlling for factual recall, relation size, and subject frequency, and is supported by experiments varying label categories, randomized mappings, and training seeds. An additional error analysis finds that fine-tuned errors align more closely with linear-probe predictions than with pretrained answer rankings. The main contribution is a controlled framework linking pretrained representation structure to subsequent knowledge transfer.

Critical strengths
  1. The experimental design clearly distinguishes between fact recall and transfer to new knowledge. By randomly relabeling the answers within existing relationships, the authors construct entirely new relationships; they then evaluate the model’s ability to generalize to previously unseen objects, without explicitly providing the original relationships or the relabeling rules. This approach establishes a practical, operational distinction between recalling a fact and leveraging the structure associated with that fact for subsequent learning. The design renders the core research question concrete and empirically testable.

  2. This study conducted experiments across 11 distinct model families and 70 different relationships, yielding robust and consistent empirical results: there is a strong correlation between pre-linearity and a model's generalization capability following fine-tuning, with Pearson correlation coefficients ranging from 0.87 to 0.92. This association remains significant even when controlling for variables such as recall rate and the number of answers.

  3. The study incorporates rigorous validation beyond core correlation analysis. it employs nested cross-validation to decouple probe fitting, layer selection, and evaluation, with preprocessing performed exclusively on the training folds. Furthermore, supplementary experiments were conducted by modifying label categories, applying random mappings, and varying random seeds. Error analysis reveals that probe-based predictions align more closely with the errors made by fine-tuned models than the rankings of answers provided by the pre-trained models themselves. Collectively, these findings substantiate that the structural properties of pre-trained representations contain meaningful information capable of predicting downstream generalization performance.

Critical weaknesses
  1. The experimental task involves a controlled category-relabeling scenario, which differs from real-world open-ended reasoning environments; therefore, the conclusions apply only to constrained transfer and should not be generalized to the broader acquisition of factual knowledge.

  2. The evaluation setup for pre-linearity and generalization is limited in scope, and sensitivity analyses for certain variables are missing; therefore, the generalizability of the conclusions regarding "active" versus "inert" knowledge remains to be verified.

  3. The study demonstrates only that linear readability can predict generalization performance; it does not confirm that this is the core underlying mechanism. It may simply be an external manifestation of other attributes, such as representation quality. Further controlled experiments are required to verify this mechanistic hypothesis.

Questions for authors

check the weakness please, I will raise my score if you can solve my questions

1 comment

We thank the reviewer for the careful reading, and for recognizing that the design separates recalling a fact from building on it, that the result holds across eleven models and seventy relations, and that the validation goes beyond the core correlation.

We respond to the third weakness. (Linear readability may only be an outward sign of another attribute, such as representation quality.) We agree. We should have measured representation quality by other means than a linear probe, and compared these measures with Pre-linearity as predictors of Generalization. If the reviewer has particular measures of representation quality in mind, we would be grateful for pointers to papers on how to measure it.

On the second weakness, could the reviewer say for which variables a sensitivity analysis is missing?

Reviewer_g1ikOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the knowledge in LLMs that can generalize under compositions. They use the subject-relation-answer triplet to model factual knowledge. For a relation rr, they fine-tune the model on a new relation g=h(r)g=h(r) that composes rr with a bijection hh with some held-out subjects that share the same answer with some other training subjects. If the model can make correct predictions on those held-out subjects for relation gg, then they call this relation rr generalizable. The authors propose that the pre-linearity of a relation is a strong indicator of this generalizibility. Here the pre-linearity measures how linear-probing reliably recovers all subject-relation-answer triplets for a fixed relation. The authors also propose some control indicators like recall accuracy, answer set size, fine-tuning subject size and subject frequency. They show that among all indicators, pre-linearity has strongest predicting power of the generalizibility of a relation.

Critical strengths
  1. The writing is very clear.
  2. I appreciate the authors' efforts on comprehensive evaluations.
  3. The research problem is interesting and important. I think this paper provides solid empirical results towards the goal of quantifying the necessary and sufficient condition for a relation to be generalizable.
Critical weaknesses

A lot of important related works are not mentioned and some of them actually share the similar linearity intuition. I assume that authors are not aware of these works. The generalization describes in this paper is almost same as the out-of-context generalization or out-of-context reasoning (OCR), which already yields heavy research literature in the community. For example, [1] theoretically proves that a one-layer transformer can be trained to perform OCR; [2,3] empirically study the mechanism of OCR. Notably, [3] shares a related intuition/observation with this paper, which they call linear correlations in knowledge composition. I recommend authors reading papers along this line and add a significant amount of dicussion about it to the paper.

[1] Yixiao Huang, Hanlin Zhu, Tianyu Guo, Jiantao Jiao, Somayeh Sojoudi, and Michael I. Jordan, Stuart Russell, Song Mei. Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformers.

[2] Jiahai Feng, Stuart Russell, Jacob Steinhardt. Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts.

[3] Letian Peng, Chenyang An, Shibo Hao, Chengyu Dong, Jingbo Shang. Linear Correlation in LM's Compositional Generalization and Hallucination.

Questions for authors

How does the generalization in this paper differ from the OCR? Especially what is the connection of this paper to [3]? I feel even though their finding of linear correlation is from a different perspective, their result seem to be consistent with this paper under a broader linear memory framework/machnism. Can you discuss on it?

1 comment

We thank the reviewer for finding the writing clear, the evaluation comprehensive, and the problem interesting and important, and for the pointers to related work.

Our paper differs from [1], [2] and [3] in the question it asks. None of them tries to predict quantitatively, from a measurement taken before fine-tuning, how well a fine-tuned relation will generalize. That is our contribution. Pre-linearity, measured on the pretrained model, predicts Generalization after fine-tuning (Pearson correlations of 0.87 to 0.92 across eleven models and seventy relations). [3] comes closest. It groups eleven handpicked relation pairs by high or low linear correlation between output logits and compares their generalization, but it does not predict generalization quantitatively. It also studies entailment, where the taught fact and a fact the model already knows entail the tested fact, not statistical generalization to unseen subjects.

We cite [2] in the introduction and in the related work. We agree that we should have discussed [1] and [3].

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.