- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper conducts a systematic investigation into property induction - the ability to generalize a trait possessed by a class of entity more readily to other classes that are taxonomically similar. The paper investigates how LLMs acquire this property by continually pretraining OSS LLMs on synthetic taxonomies (generated using Wuggy to avoid pretraining confounds). The paper then measures whether the probability mass the trained LLM assigns to an entity having within-category traits exceeds the mass it assigns to the entity having control cross-category traits - called the 'induction advantage'.
They find that multiple families of OSS LLM readily acquire this ability, but only if shared class features/taxonomical hierarchy is made clear during the synthetic pre-training phase. They also demonstrate that LLMs further along in pretraining exhibit larger induction advantages, and surprisingly that, that larger models exhibit smaller induction advantages at matched data. The taxonomy is even directly reconstructable from model latents via clustering algorithms. The paper hypothesizes that this may be due to larger models memorizing the facts faster, thus not needing to create a general taxonomy.
The paper also turns to in-context learning, and evaluates how models can learn taxonomies in context - when trained only with the shared properties of the taxonomy, models can still recover the inductive advantage from in-context leaf-specific properties. They additionally show models can learn taxonomies that go against their training prior in-context. They also show that learning a second taxonomy in pretraining after the first is harder - indicating that in-context adaptability does not apply pre-training adaptability.
The paper concludes by synthesizing these experiments into a holistic view of property induction, and how it emerges from simple next-token pretraining.
Critical strengths
- The paper is extremely well-controlled, and the controls are clear and easy to understand. The 'induction advantage' is clearly defined and took me only a minute or so to grasp - the difference in logprob space between the within-category trait and the control distractor traits. Similarly, the demonstration that shared properties mediate the induction advantage is also well-controlled. This applies even to the smaller experiments - worse performance on taxonomies learned second is compared to learning those taxonomies directly, to ensure the generated taxonomies aren't accidentally more difficult.
- The paper carefully replicates the exact pretraining mixtures of Pythia/Olmo used, with synthetic facts interspersed at a fixed percentage. This semi-realistically simulates organic encountering of such facts during pre-training, instead of post-hoc fine-tuning, and makes the author's arguments especially strong.
Critical weaknesses
None
Questions for authors
Clarification Questions:
- You state in the text that "We can see that as in-context shots accumulate, inductive generalization to the in-context closest target E3 overtakes generalization to the in-weights target E2, and the model generalizes according to the context taxonomy (Figure 7, bottom)." But in the provided figure, E3 always exceeds E2 at all shot counts - both rise in probability, but order is consistent. There isn't a visible 'crossing over' like the text suggests. Perhaps it is off the y-axis, at 0, 1, or 2 shots?
Suggestions for Future Work:
- You only test hierarchies of size D = 3. How might performance change for smaller or larger hierarchies? Might the advantages of larger models only become clear as D >> 3?
- Perhaps induction advantage is U-shaped - such that models initially get worse, but larger ones then get better (for instance, GPT-4 on hindsight neglect). Obviously, GPT-4 scale models are out of scope here, but there are larger Pythia models that could be tested - if compute allowed.
- Is the link between memorization ability and lack of induction advantage causal? If you somehow impaired the memorization ability of one of the larger models during synthetic pretraining (perhaps by training only attention layers, or introducing some form of dropout), would induction advantage increase?
5 comments
Rebuttal (2/2)
Suggestion for Future Work 1: Smaller or larger hierarchies.
We agree that mapping the memorization/generalization balance as a function of the size of the taxonomy or the data budget is an important next step. We see two effects of a deeper () and/or wider tree, which go in opposite directions.
First, the number of entity-property facts grows very quickly. This makes memorizing each fact separately more costly, so representing entities through their categories could become more attractive even for larger models. Under our memorization-generalization trade-off hypothesis (Section 3.3), we would expect a tipping point in the ratio of model size to the number of facts, after which models favor inductive generalization right away. In this regime, model scale would likely help, because a larger model could separate more fine-grained categories. The disadvantage of larger models that we observe may therefore shrink or turn into an advantage.
Second, for a fixed model size, a larger taxonomy means that more, finer categories must be separated. If the share of synthetic data in each batch is also kept fixed, each fact is seen less often. We therefore expect the qualitative effect to stay the same (generalization within categories stronger than across categories), but the induction advantage could be lower than for .
Which of the two effects dominates at a given model size is an empirical question. We did not have the compute to test larger hierarchies during the rebuttal. Our new experiment with a larger data budget (Appendix D.8, Figure 17) gives indirect support for the first effect: it shows that larger models can reach stronger taxonomy-aligned generalization than smaller ones once the data supports it. The limitations in Appendix A already listed larger taxonomies as future work (first paragraph). We have now added a paragraph (the fifth) on the reviewer's specific point that the disadvantage of larger models may shrink or turn into an advantage for .
Suggestion for Future Work 2: U-shaped induction advantage.
This is a very interesting hypothesis, which we cannot rule out. Within the range and setting we tested, the trend is monotonic (Pythia 70M to 2.8B, and OLMo 2 1B to 7B). The new decomposition of induction advantage in Appendix D.7 also shows that for Pythia 2.8B, the probe property fluctuates around and below the uniform share of the probability mass, so there is no sign of a recovery yet. But a recovery in larger models is possible, similar to model-wise double descent (Nakkiran et al., 2020). We now mention testing this hypothesis about the U-shaped induction advantage in the limitations in Appendix A.
Preetum Nakkiran et al. Deep Double Descent: Where Bigger Models and More Data Hurt. ICLR 2020.
Suggestion for Future Work 3: Relationship between memorization and induction advantage.
We cannot yet confirm that the link between memorization and induction advantage is causal: in the current form, our results on this relationship are correlational. Specifically, we show that larger models memorize faster (Figure 13) and give a smaller share of their probability to the probe property, leading to lower induction advantage (Figure 16), but we do not intervene on memorization to test the causal relationship. In Section 3.3, we present the link between memorization and induction as a hypothesis, and in the Conclusion, we state it as one possible explanation. Additionally, we now say explicitly in the limitations (Appendix A) that the shown relationship is correlational and not an established cause. The interventions that the reviewer suggests (training only the attention layers, or adding dropout during the injection) are promising ways to test causality, and we now list them as future work in Appendix A.
Thank you for the clarification! The caption makes more sense, and this resolves my concern.
Rebuttal (1/2)
We thank the reviewer for the positive review, for catching the mismatch between the text and Figure 7, and for the insightful suggestions for future work. We are glad that the reviewer found the paper “extremely well-controlled” and appreciate the comment that the semi-realistic setup of synthetic data injection makes our arguments especially strong. We address each of the reviewer’s questions separately below.
The reviewer is right in pointing out this disagreement between the text and the figure. In Figure 7, there is no crossover, and our word “overtakes” was not used correctly. It is also important to note that the models in this experiment are trained only on the shared properties, so at zero shots, which the reviewer mentioned, the models have never seen the probe property P. Therefore, there is nothing to generalize yet, and the probability of P for both the in-context and in-weights closest entities starts very low (below the y-axis range). With more shots, the probability of P rises for both entities, because the models learn to reuse a property given in the prompt. But it is higher for the in-context closest entity E3 than for the in-weights closest entity E2 from the first shot on, and the order stays the same at every number of shots. The context taxonomy therefore wins as soon as it is present, not gradually as the original text suggested. We rewrote the sentence in Section 3.5 accordingly, removing this disagreement.