- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how language models learn and store knowledge that changes, and how that is affected by the way data is ordered during pretraining. The paper compares a training strategy where facts that change are ordered temporally in the training data (ORDERED) with a setting where the changing facts are randomly ordered (SHUFFLE). In a small-scale synthetic setting, the paper finds that ORDERED training data causes facts to be stored in individual slots that are easily updated with little effect on other facts, whereas SHUFFLE training data causes facts that generally move together (e.g. athletes that switch teams together) to be stored together in class-based storage and to generally be updated together. The paper then briefly investigates OLMo-2-7B, a released language model whose pretraining is analogous to SHUFFLE. The authors find that an edit to OLMo-2-7B of one fact moves 9.8% of the other facts sharing its relation and answer and that classes whose answers changed more across pretraining checkpoints are more strongly coupled, hinting at class-based storage which matches the results in the synthetic setting. However, a full analysis is not possible as there is no olmo-equivalent of a model trained on ORDERED data, so this result is correlational and cannot show that the coupling arises from how OLMo observed changing facts, as the authors acknowledge.
Critical strengths
The synthetic setting is comprehensive and tests counterfactual hypotheses to make sure findings are correct. It is well explained and easily understandable. I also think the finding that the order of training data affects how knowledge is stored is novel and interesting, even if it is difficult to thoroughly test and conclude that the same patterns exist in large pre-trained models like OLMo (OLMo is the SHUFFLE equivalent of the synthetic setting but no ORDERED equivalent exists).
Critical weaknesses
- the OLMo setting is only evaluated on one model; it is unclear whether the results reproduce on models of e.g. different sizes (though these are available; e.g. OLMo 2 is released at 1B, 7B, 13B and 32B).
- There is only correlational evidence connecting the synthetic and OLMo setting and the experimental setup is not the same in the two settings. The OLMo setting cannot do an actual comparison of ORDERED versus SHUFFLE as the pretrained setting does, as OLMo is the SHUFFLE analogy but no ORDERED analogy exists. This is a problem because the finding that coupling of facts exists in OLMo as SHUFFLE would predict doesn't mean that a model similar to OLMo trained in ORDERED fashion would show facts stored in individual slots, so the analysis is incomplete. Moreover, in the synthetic setting, classes are defined as groups of facts that have changed together throughout training and shows that without this shared history there is no coupling. However, in the OLMo setting, classes are defined as groups of facts that share the same relation and answer without considering if facts changed together during training.
Questions for authors
- what is representation rank? What is participation ratio? These terms are not defined. It is unclear to me what exactly figure 5a/the respective paragraph in 4.3 is measuring. For example, which activations are used? At which token position? Over what vectors? Is your inference that low rank activations imply that models are not memorizing shifting facts? That in itself is not obvious to me?
- in an attempt to make the OLMo and synthetic setting more similar, can you repeat the OLMo experiment where you edit facts by finetuning with facts from groups where facts changed together across checkpoints?
2 comments
Thank you for the response and the additional experiments. The clarifications on rank resolve my concern. On OLMo co-movement, my concern is partially addressed: my question concerned groups whose facts changed together across model checkpoints, while the new experiment examines groups whose facts changed together in the real world. This does not establish whether the model learned these facts as changing together during pretraining. Still, the new experiment provides useful evidence that updates spread more readily among facts that changed together historically.
My concern about evaluation on only one pretrained model is unaddressed: it is unclear whether the OLMo results reproduce across model sizes. Another remaining limitation is that there is only correlational evidence connecting the synthetic and OLMo settings, though I appreciate that this is difficult to resolve as no ORDERED-OLMo equivalent exists. I retain my rating of weak accept.
Response to Reviewer
Thank you for your insightful feedback. We were encouraged that they you found our discoveries to be novel and interesting, and the synthetic setting to be comprehensive and well-tested.
Clarifications on Rank
We apologize for the lack of clarity in the writing and for omitting an explicit formula of the participation ratio. In all parts of the paper, we measure the representation rank via the participation ratio.
Given a set of sampled activations, whose empirical covariance matrix has eigenvalues λ1,...,λn, the participation ratio is defined as ∑i=1n(λi)2(∑i=1nλi)2. This is a measure of how concentrated the variance of the data is.
In calculating this participation ratio, we take the MLP-input activations of the model directly preceding the position of the answer token. In our main paper results, we used the activations before the final MLP block, but we provide additional results across layers in Appendix J. As in prior formulations of key-value fact storage, this represents the key vector through which the model retrieves its stored answer.
We do not intend to put forward any inference about whether the model has memorized using these representational results. For clarity, all models that we test here have 95%+ accuracy on the final time-step shifting facts, which suggests that the model does memorize them (regardless of the rank). The role of participation ratio in our analysis is primarily as a metric of how facts are stored in the model internally.
Intuitively, a low participation ratio means that most of the variance across shifting facts lies along a few directions shared by facts in the same class, suggesting less indvidualized representations. A high participation ratio means that each fact also varies along directions of its own, indicating more individualized representations.
OLMo Co-Movement
We have added a version of this experiment to the revised Appendix K, where we show that updates spread further among facts that changed together in the real world. We consider 24 classes of facts, each group of which changed together during a known hisotrical event. Some examples include a group of AFL teams that joined the NFL together in 1970 or the cities of East Germany in 1990. After an update to part of a group, 71.4% of the remaining members adopted the new answer, compared with 60.2% of classmates that reached the same answer through a different event (+11.1 points, 95% interval [1.5, 19.8]). This analysis, by design, controls for potential confounds of relation type, fact popularity, and difficulty of the edit and supports our findngs of co-movement classes in a pre-trained model.