- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether the linear readability of pretrained relational knowledge predicts generalization after fine-tuning. The authors first measure the linear readability of selected relations, then synthesize new relations by randomly relabeling their answers, fine-tune the models on a subset of subjects, and test them on held-out subjects. On 11 models, they show that pre-linearity strongly correlates with the generalization performance they define.
Critical strengths
- The authors validate their findings across 11 models and 70 relations drawn from multiple data sources.
- The paper describes its methodology and experimental settings in detail, supporting the reproducibility of the results.
Critical weaknesses
- I have concerns about the paper's contribution. All experiments are conducted in a toy, synthetic setting. The analysis is largely descriptive and provides limited insight into the underlying reasons, such as why pre-linearity would associated with generalization. The paper also offers little practical guidance on how this property could be applied in actual model training.
- I also have concerns about the rigor of the experimental conclusions, the interpretation of the observed results, and what the authors regard as generalization. I elaborate on these concerns in the Questions section.
Questions for authors
- Regarding the experimental setup and interpretation:
- Using the authors' Italy and Munich example, how can we establish that the model answers Munich → crab by learning the Europe → crab mapping or by using the fact that both Italy and Munich are in Europe? Could there be alternative explanations? For example, the model might simply rely on similarity between subjects: if Italy is the training subject most similar to Munich, it might assign Munich the same label, crab. Would other subjects associated with Italy, such as pizza or Galileo, also be mapped to crab?
- My previous point concerns both how generalization is defined in this paper and how it is inferred from the experiments. If the authors attribute the Munich → crab prediction to the model's knowledge of the location-to-continent relation, could this generalization be tested more directly? For example, could they test whether the model has learned the equivalence between Europe and crab?
- Generalization should also be distinguished from overgeneralization. If the model has indeed learned that only locations in Europe should be assigned crab and only locations in Asia should be assigned zebra, would it avoid predicting crab or zebra for a city on a continent not covered by the experiments?
- Lines 254–255: Why is the query template needed when probing the representation of Paris? In a causal language model, shouldn't the representation at Paris be unaffected by the subsequent text, such as "is located in"?
- The Pre-linearity paragraph in Section 3 would be easier to follow with pseudocode outlining the procedure.
2 comments
Most of my concerns have not been addressed. I maintain my original rating.
We thank the reviewer for recognizing that the findings are validated across 11 models and 70 relations drawn from multiple data sources, and that the detailed description of the methodology supports reproducibility.
We respond to the main concern, that the experiments use a toy, synthetic setting.
Only the new answers are synthetic. The models are pretrained language models, and the relations are real relations they learned in pretraining. We invent the new answers because the question requires it. To measure how a model generalizes knowledge it has just been taught, that knowledge must be certain to be absent from pretraining, and its tie to existing knowledge must be controlled. Real facts guarantee neither.
This design is established practice at ICLR and other top venues. Berglund et al. [1] establish the reversal curse by fine-tuning language models on invented facts about fictitious people. Feng et al. [2] fine-tune on facts about fictitious people. Huang et al. [3] follow this design. Jain et al. [4] study the effects of fine-tuning in synthetic, controlled settings.
[1] Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, Owain Evans. The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A". ICLR 2024.
[2] Jiahai Feng, Stuart Russell, Jacob Steinhardt. Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts. ICML 2025.
[3] Yixiao Huang, Hanlin Zhu, Tianyu Guo, Jiantao Jiao, Somayeh Sojoudi, Michael I. Jordan, Stuart Russell, Song Mei. Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformers. NeurIPS 2025.
[4] Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, et al. Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks. ICLR 2024.