acceptodds
Under review as a conference paper at ICLR 2027

Latent Taxonomies: How Property Induction Develops in Language Model Pre-training

100 views

Abstract

When told that robins have sesamoid bones, people are far more prone to extend the property to sparrows than to whales. This behavior, called category-based property induction, is well documented in humans, but whether and how it develops in language models trained only to predict the next token is unclear. To shed light on this, we construct a hierarchical taxonomy of synthetic entities grouped by the properties they share. We then inject facts from this latent taxonomy into the continued pre-training of Pythia and OLMo 2 models from 70M to 7B parameters, and track the development of inductive behaviors. We find that taxonomically aligned induction emerges from next-token prediction alone: a property learned for one entity transfers more strongly within its taxonomic category than across categories. Internally, entities that share a deeper category become more similar in activation space, and clustering recovers the full taxonomic tree from entity representations. Contrary to the usual expectation that scale helps, we find that induction weakens with model size when the data budget is fixed, suggesting that larger models memorize individual facts rather than compress them into shared structure. Induction also strengthens with more pre-training before the taxonomy injection, transfers to novel in-context properties, and can be overridden by a latent taxonomy implied by the prompt. Together, these results show that a model's inductive generalization reflects the latent organization of its training data, with implications for data curation and for predicting how far a learned fact will spread.

Primary area:
foundation or frontier models, including LLMs
Keywords:
property induction, generalization, language models, pre-training, learning dynamics, representational geometry

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 4.00 over 3 reviews, above 99% of the others.
Reviewer_HsDpOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper conducts a systematic investigation into property induction - the ability to generalize a trait possessed by a class of entity more readily to other classes that are taxonomically similar. The paper investigates how LLMs acquire this property by continually pretraining OSS LLMs on synthetic taxonomies (generated using Wuggy to avoid pretraining confounds). The paper then measures whether the probability mass the trained LLM assigns to an entity having within-category traits exceeds the mass it assigns to the entity having control cross-category traits - called the 'induction advantage'.

They find that multiple families of OSS LLM readily acquire this ability, but only if shared class features/taxonomical hierarchy is made clear during the synthetic pre-training phase. They also demonstrate that LLMs further along in pretraining exhibit larger induction advantages, and surprisingly that, that larger models exhibit smaller induction advantages at matched data. The taxonomy is even directly reconstructable from model latents via clustering algorithms. The paper hypothesizes that this may be due to larger models memorizing the facts faster, thus not needing to create a general taxonomy.

The paper also turns to in-context learning, and evaluates how models can learn taxonomies in context - when trained only with the shared properties of the taxonomy, models can still recover the inductive advantage from in-context leaf-specific properties. They additionally show models can learn taxonomies that go against their training prior in-context. They also show that learning a second taxonomy in pretraining after the first is harder - indicating that in-context adaptability does not apply pre-training adaptability.

The paper concludes by synthesizing these experiments into a holistic view of property induction, and how it emerges from simple next-token pretraining.

Critical strengths
  • The paper is extremely well-controlled, and the controls are clear and easy to understand. The 'induction advantage' is clearly defined and took me only a minute or so to grasp - the difference in logprob space between the within-category trait and the control distractor traits. Similarly, the demonstration that shared properties mediate the induction advantage is also well-controlled. This applies even to the smaller experiments - worse performance on taxonomies learned second is compared to learning those taxonomies directly, to ensure the generated taxonomies aren't accidentally more difficult.
  • The paper carefully replicates the exact pretraining mixtures of Pythia/Olmo used, with synthetic facts interspersed at a fixed percentage. This semi-realistically simulates organic encountering of such facts during pre-training, instead of post-hoc fine-tuning, and makes the author's arguments especially strong.
Critical weaknesses

None

Questions for authors

Clarification Questions:

  • You state in the text that "We can see that as in-context shots accumulate, inductive generalization to the in-context closest target E3 overtakes generalization to the in-weights target E2, and the model generalizes according to the context taxonomy (Figure 7, bottom)." But in the provided figure, E3 always exceeds E2 at all shot counts - both rise in probability, but order is consistent. There isn't a visible 'crossing over' like the text suggests. Perhaps it is off the y-axis, at 0, 1, or 2 shots?

Suggestions for Future Work:

  • You only test hierarchies of size D = 3. How might performance change for smaller or larger hierarchies? Might the advantages of larger models only become clear as D >> 3?
  • Perhaps induction advantage is U-shaped - such that models initially get worse, but larger ones then get better (for instance, GPT-4 on hindsight neglect). Obviously, GPT-4 scale models are out of scope here, but there are larger Pythia models that could be tested - if compute allowed.
  • Is the link between memorization ability and lack of induction advantage causal? If you somehow impaired the memorization ability of one of the larger models during synthetic pretraining (perhaps by training only attention layers, or introducing some form of dropout), would induction advantage increase?
5 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We thank the reviewer for the positive review, for catching the mismatch between the text and Figure 7, and for the insightful suggestions for future work. We are glad that the reviewer found the paper “extremely well-controlled” and appreciate the comment that the semi-realistic setup of synthetic data injection makes our arguments especially strong. We address each of the reviewer’s questions separately below.


Clarification Question: Disagreement between the text and Figure 7.

The reviewer is right in pointing out this disagreement between the text and the figure. In Figure 7, there is no crossover, and our word “overtakes” was not used correctly. It is also important to note that the models in this experiment are trained only on the shared properties, so at zero shots, which the reviewer mentioned, the models have never seen the probe property PP. Therefore, there is nothing to generalize yet, and the probability of PP for both the in-context and in-weights closest entities starts very low (below the y-axis range). With more shots, the probability of PP rises for both entities, because the models learn to reuse a property given in the prompt. But it is higher for the in-context closest entity E3 than for the in-weights closest entity E2 from the first shot on, and the order stays the same at every number of shots. The context taxonomy therefore wins as soon as it is present, not gradually as the original text suggested. We rewrote the sentence in Section 3.5 accordingly, removing this disagreement.

AuthorsOpenReview ↗

Rebuttal (2/2)

Suggestion for Future Work 1: Smaller or larger hierarchies.

We agree that mapping the memorization/generalization balance as a function of the size of the taxonomy or the data budget is an important next step. We see two effects of a deeper (D>3D>3) and/or wider tree, which go in opposite directions.

First, the number of entity-property facts grows very quickly. This makes memorizing each fact separately more costly, so representing entities through their categories could become more attractive even for larger models. Under our memorization-generalization trade-off hypothesis (Section 3.3), we would expect a tipping point in the ratio of model size to the number of facts, after which models favor inductive generalization right away. In this regime, model scale would likely help, because a larger model could separate more fine-grained categories. The disadvantage of larger models that we observe may therefore shrink or turn into an advantage.

Second, for a fixed model size, a larger taxonomy means that more, finer categories must be separated. If the share of synthetic data in each batch is also kept fixed, each fact is seen less often. We therefore expect the qualitative effect to stay the same (generalization within categories stronger than across categories), but the induction advantage could be lower than for D=3D=3.

Which of the two effects dominates at a given model size is an empirical question. We did not have the compute to test larger hierarchies during the rebuttal. Our new experiment with a larger data budget (Appendix D.8, Figure 17) gives indirect support for the first effect: it shows that larger models can reach stronger taxonomy-aligned generalization than smaller ones once the data supports it. The limitations in Appendix A already listed larger taxonomies as future work (first paragraph). We have now added a paragraph (the fifth) on the reviewer's specific point that the disadvantage of larger models may shrink or turn into an advantage for D≫3D \gg 3.


Suggestion for Future Work 2: U-shaped induction advantage.

This is a very interesting hypothesis, which we cannot rule out. Within the range and setting we tested, the trend is monotonic (Pythia 70M to 2.8B, and OLMo 2 1B to 7B). The new decomposition of induction advantage in Appendix D.7 also shows that for Pythia 2.8B, the probe property fluctuates around and below the uniform share of the probability mass, so there is no sign of a recovery yet. But a recovery in larger models is possible, similar to model-wise double descent (Nakkiran et al., 2020). We now mention testing this hypothesis about the U-shaped induction advantage in the limitations in Appendix A.

Preetum Nakkiran et al. Deep Double Descent: Where Bigger Models and More Data Hurt. ICLR 2020.


Suggestion for Future Work 3: Relationship between memorization and induction advantage.

We cannot yet confirm that the link between memorization and induction advantage is causal: in the current form, our results on this relationship are correlational. Specifically, we show that larger models memorize faster (Figure 13) and give a smaller share of their probability to the probe property, leading to lower induction advantage (Figure 16), but we do not intervene on memorization to test the causal relationship. In Section 3.3, we present the link between memorization and induction as a hypothesis, and in the Conclusion, we state it as one possible explanation. Additionally, we now say explicitly in the limitations (Appendix A) that the shown relationship is correlational and not an established cause. The interventions that the reviewer suggests (training only the attention layers, or adding dropout during the injection) are promising ways to test causality, and we now list them as future work in Appendix A.

Thank you for the clarification! The caption makes more sense, and this resolves my concern.

Reviewer_or51OpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors insert synthetic data during pretraining runs at checkpoints for open source models (Pythia & Olmo) to see how LMs learn hierarchical structure. The synthetic data contains a latent hierarchy, where each node in the tree shares properties with its neighbors. The authors ask how well the model learns to share these properties across related entities, whether the taxonomy is reflected in the model's learned representations, how model size affects generalization across the taxonomy, and what happens when in-context information conflicts with the pretrained hierarchy.

Critical strengths

The question of how LLMs generalize is still open, and this paper provides compelling evidence of models learning structure in a synthetic, but still realistic, setting. We see that models encode the known hierarchical structure directly in their representations, and that there seems to be some relationship between model size/training data and the tendency to directly learn the hierarchy vs. broadly learn the task + memorize individual facts.

Critical weaknesses

Weakness:

  • I'm concerned with the "induction advantage" setup for the model size experiments. It looks like models at different classes have similar log probs for the correct entity showing similar amounts of memorization. However, it seems like larger models giving more mass to the synthetic tokens in general would hurt their induction advantage — but this may actually be a correct form of generalization to recognize that the synthetic tokens go with the synthetic task (compared to, say, baseline tokens)?

Minor/clarity suggestions:

  • Section 2.2: The section beginning with "To test whether LMs trained on these shared properties..." is hard to parse — and I think this is the key idea of the experimental setup. It seems like you're using "target" to mean both the entity seen with the property and its sibling? But the log prob you're actually measuring is on the adverb word "bragely"? Which could also be thought of as the "target"? I think Figure 1 makes sense, so I would just rewrite this section.
  • Section 2.3: Move the footnote for log p (property | entity) to the end of the sentence — it looks like it means "squared"
  • Figure 4: The lines going from the x-axis to each dot are confusing. Maybe remove those unless there's a reason for them? They cover the confidence intervals and make the graph hard to read.
  • Conclusion: This reads like LLM generated text. Probably worth rewriting this.
Questions for authors

Suggestions/Questions:

  • Am I correct that the higher levels of the hierarchy ("organisms", "animals") are never trained? That all the synthetic data is on the leaves of the hierarchy tree? (If this is not stated explictly in the main body, I would recommend doing so)
  • In Figure 2, it seems like the log probs for everything plummet near training step 10^4? I don't really understand what's happening here. Is this because the synthetic data is pulled from training here?
  • Have you run any more experiments with a larger data budget for larger models? You mention this in the future work & limitations section, and I think this would be helpful to clarify what's actually going on with model size.
  • Did you try anything to understand how the hierarchy is organized in the representations? Did you plot the learned clusters? Try any other clustering methods?
5 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

We thank the reviewer for the careful reading and the concrete suggestions. In response, we ran one new experiment (model-size comparison with a four-times larger data budget), added two new analyses (a decomposition of the induction advantage into total probability mass and its split, and a closer look at how the hierarchy is organized in the representations), and rewrote the parts of the paper the reviewer found unclear. We discuss each in detail below.


Weakness: Concern with induction advantage.

This is a good point, and we now address it in the new Appendix D.7 (Figures 15 and 16), and provide a short summary of it in Section 3.3.

First, as for the metric itself, the induction advantage is a difference of log-probabilities: log⁡p(P)\log p(P) minus the mean of log⁡p(Ci)\log p(C_i). It therefore depends only on the ratio between the probe property and the control properties. If a larger model gives more mass to all synthetic properties alike, both terms increase by the same amount and the induction advantage does not change. So learning that “synthetic tokens go with the synthetic task” cannot lower the metric by itself. The metric can therefore drop only if the additional mass goes more to the control properties CiC_i from other categories than to the within-category property PP. We confirm that this is the case in a new analysis with the Pythia family. Specifically, we split the next-token probabilities for the held-out (formerly “target”) entity into two parts:

  1. Total mass on the probe and all control properties, p(P)+∑ip(Ci)p(P) + \sum_i p(C_i) (Figure 15). Here, we find that the reviewer’s intuition is correct: this total synthetic-property mass grows with model size, from roughly one half for the 70M model to roughly two thirds or more for 410M and larger models.
  2. Share of this mass that each property receives (Figure 16). Here the trend in terms of model size goes the other way: the 70M model gives about half of the mass to the probe property PP alone. For 160M and 410M, the share of PP is lower but still clearly above the uniform share of 1/7. For 2.8B, it is at or below the uniform share, and later in the injection it is below the share of the control properties.

So we agree with the reviewer that larger models learn more strongly that the synthetic properties go with the synthetic entities, and that this may be called a correct generalization about the overall task. However, they spread this mass more evenly, and for 2.8B the probe property no longer gets more than the control properties. This result that the inductive generalization of larger models follows the taxonomy less closely is then reflected in the lower induction advantage. Figure 16 also shows that for 160M and 410M, the split is graded by taxonomic distance: control properties of entities that share the L2 category with the held-out entity get more mass than those of entities that share only the root category L1.

Altogether, this new analysis shows where the lower induction advantage of larger models comes from: not from the amount of probability they assign to the synthetic properties overall (which is in fact higher), but from how they divide it, with a smaller share going to the within-category property PP and a larger share to the properties of more distant categories.


Minor/clarity suggestion 1: Section 2.2 (use of 'target').

We rewrote this paragraph and no longer use the word “target”, which one might interpret as an entity or the scored tokens. Instead, the revised text separates three things: (1) the source entity, which is the only entity trained with the probe property PP; (2) the held-out entity, which is the source’s same-category sibling and is never trained with PP but only with the shared properties; and (3) the tokens we actually score, which are the adverb after the property verb (e.g., bragely vs. rourly after the prompt ruper chubbles). For consistency, we use the same terms in the definitions of memorization and induction in Section 2.3 and in the appendix. We also removed “target” from Section 3.5 and Figure 7, which now refer to the “in-weights closest entity” and the “in-context closest entity”.


Minor/clarity suggestion 2: Section 2.3 (footnote placement).

We thank the reviewer for noticing this. We fixed it in the revised submission, with the footnote now at the end of the sentence.


Minor/clarity suggestion 3: Figure 4 (vertical lines).

We removed the vertical lines from the x-axis to the individual dots in Figure 4 and Figures 5 and 13, which used the same style. The spread across templates is now easier to see.


Minor/clarity suggestion 4: Conclusion.

We rewrote the conclusion in the revised submission. It is now split into two paragraphs: the first states what we did and that induction emerges from next-token prediction, and the second lists the factors that change its strength.

AuthorsOpenReview ↗

Rebuttal (2/3)

Question 1: Higher levels of the taxonomic hierarchy.

Yes, this is correct: the higher levels of the hierarchy (“organisms”, “animals”) are never trained. All synthetic training sentences pair a leaf entity with one of its properties. The category nodes have no names and never appear in the text. The higher levels of the hierarchy exist only as properties shared by all leaves below a node (in the Figure 1 analogy, the model would never see “animal”, only that dog and cat both “have muscles”). This was stated only in Appendix C.2. We now state it explicitly in Section 2.2.


Question 2: Figure 2 (drop near step 10410^4).

Yes, the plummeting of log probs near training step 10410^4 is because the injection of synthetic data stops at step 5,000, and training then continues on the background corpus alone until step 10,000 (to study forgetting). The x-axis is logarithmic, so step 5,000 is close to the right edge of each panel near training step 10410^4, and the blue bar at the bottom shows the injection steps. The sharp drop is the forgetting that we show in Figure 12 (Appendix D.4). We now say this in the caption of Figure 2 and in the text of Section 3.1.


Question 3: Larger data budget for larger models.

We ran this experiment with a larger data budget for the rebuttal (new Appendix D.8, Figure 17). Specifically, we repeated the comparison of Pythia 160M and Pythia 1B with a four-times larger data budget: 20,000 steps with data injection, keeping everything else unchanged from the original experiments.

Early in training, the new Figure 17 agrees with Figure 5: the 160M model reaches an induction advantage of about 1.6 within the first 1,000 steps, and the 1B model is clearly below it for the first few thousand steps. However, the 1B model then closes the gap: the induction advantage curves of the two models meet after roughly 5,000 to 7,500 steps, and from about step 8,000 on, the 1B model has the higher induction advantage (about 2.3 vs. about 2.0 at step 20,000).

So the lower induction advantage of larger models at our default budget does not mean that they cannot generalize along the taxonomy. They acquire this generalization more slowly and need more exposure, and with enough exposure the larger model reaches and exceeds the smaller one. This fits our memorization explanation: a larger model first fits the injected facts by memorizing them and comes to rely on the shared categories only later, potentially due to regularization (either explicit in the form of weight decay or implicit from the optimization dynamics). We revised the claims to make them more precise: in the abstract and the introduction, we now say that scale hurts induction under a fixed and limited data budget, and in Section 3.3, we say that the larger model catches up with more exposure. This new comparison covers only two sizes of one model family, so we left a full sweep over data budget and model size for future work (Appendix A).

AuthorsOpenReview ↗

Rebuttal (3/3)

Question 4: Organization of the taxonomic hierarchy in model representations.

We thank the reviewer for these questions. We added a new appendix section with PCA plots of the learned clusters in Pythia 160M and a comparison of six clustering methods.

PCA of entity representations. For every seed, we plot the first two principal components of the entity representations, before and after injection. We use the hidden state of the last entity token at the layer analysed in the paper, averaged over the many sentence contexts in which entities appear in the templates, and over the checkpoints saved during taxonomy injection. Colours and convex hulls mark the two L2 categories, and markers and connecting lines mark the four lowest-level (L3) categories.

As Figure 11 in Appendix D.3 of the revised submission shows, injecting the synthetic data makes the structure of internal entity representations more aligned with that of the ground-truth latent hierarchy. Specifically, for one of the seeds (seed 0), the two L2 categories separate along PC1 after injection, and each L3 pair is well separated from the other pair. In contrast, before injection, they overlap. In seeds 1 and 2, the structure is partial: in seed 1, the L3 pairs and most of the L2 split appear, but one entity is in the other category. In seed 2, the L3 pairs move together, but the L2 categories partially overlap. It is important to note that the two principal components together explain only about 35 to 43% of the variance, so the projection shows a limited part of the geometry, and structure that is not visible here can still be present in the remaining components.

Other clustering methods. Besides Matching Affinity Clustering (MAC) used in the paper, we evaluated four standard agglomerative linkages: UPGMA, complete, WPGMA, and Ward, and standard k-means (a flat method). For each method, we cut the recovered tree into four clusters and score them with the adjusted Rand index (ARI) against the four ground-truth L3 categories (0 = chance, 1 = identical). Finally, we compare each score with a permutation null (300 random relabelings of the taxonomy) and with the model before injection.

As Table 2 in Appendix D.3 (reproduced below) shows, every clustering method recovers the lowest-level categories (L3) well above the permutation null and above the model before injection. This shows that the representations reflect the latent taxonomy, but only in part: the scores are well below 1, in line with the Pythia 160M dendrogram in Figure 10 (Appendix D.2), which also does not match the ground truth exactly. Furthermore, standard agglomerative methods and k-means do as well as MAC or better (ARI 0.45-0.52), so recovering the taxonomy from the representations does not depend on MAC.

Table: ARI against the four L3 categories in Pythia 160M (mean across seeds).

BeforeAfter
MAC (paper).07.42
UPGMA.05.52
Complete linkage.07.45
WPGMA.05.52
Ward.07.45
k-means.07.45
Permutation null.00.00
Reviewer_4AiZOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies if and how a particular kind of inference over taxonomies, namely property induction, develops over LM pretraining. Property induction assumes that the closer two classes are in a taxonomy, the more likely it is that they share properties.

The authors devise a clever training setup in which synthetic taxonomy data is injected natural pretraining data. This semi-synthetic data allows isolating the phenomenon of interest, i.e., tracking how the model learns the taxonomy, over the course of an otherwise natural pretraining.

Using this training setup, the paper finds, among others, that

  • smaller LMs exhibit a stronger propensity towards property induction than larger ones, plausibly because larger models have more capacity for memorization and hence less pressure towards inference;
  • taxonomic distance is correlated with representational similarity, to the degree that the taxonomy can be recovered by hierarchical clustering of the entity representations; and that
  • taxonomies learned during pretraining can influence in-context induction.
Critical strengths
  1. The hybrid synthetic-real pretraining setup will likely be useful in general
  2. The findings on when and how property induction develops are interesting and will likely inform future work
Critical weaknesses

none

Questions for authors

none

1 comment

We thank the reviewer for their positive feedback and comments on the paper's potential to positively influence future work. We are particularly grateful that the reviewer pointed out that our hybrid synthetic-real pretraining setup, in which synthetic taxonomy data is injected into natural pretraining data, will likely be useful in general. We also appreciate the reviewer's comment that our findings on the development of property induction are interesting and will likely inform future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.