acceptodds
Under review as a conference paper at ICLR 2027

What Does a Model Learn from a Changing World?

Abstract

Models absorb world knowledge during pretraining, but they do not see a static snapshot of the world: pretraining data spans many time periods, showing facts change over time. In a controlled setting, we first ask whether models simply memorize each time period separately, or learn how knowledge evolves across them. We find the latter: compared to a model trained on a single time period, models trained on multiple periods (a) better preserve historically constant knowledge and (b) more efficiently learn updates that follow historical patterns. We then find that the order in which facts are presented determines how they are stored. When facts are presented chronologically, they are stored in individual slots, so an update changes only the facts it trains on. When facts from different time periods are instead shuffled together, models use class-based storage: facts that have changed together, such as athletes who switch teams together, share a representation, so an update to one of them spreads to the others. We find evidence of class-based storage in OLMo-2-7B, whose pretraining data, like our shuffled setting, mixes documents from different time periods. Beyond the known tendency of updates to spread among facts that share a relation and answer, we find correlational evidence that updates spread further in groups of facts whose answers changed more during pretraining. Our results show that models adapt to the changes they observe during training, and that the order in which they observe them shapes how, suggesting pretraining data ordering as a way to influence how models absorb new information.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Continual Learning, Factuality, Training Dynamics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AZL7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how language models learn and store knowledge that changes, and how that is affected by the way data is ordered during pretraining. The paper compares a training strategy where facts that change are ordered temporally in the training data (ORDERED) with a setting where the changing facts are randomly ordered (SHUFFLE). In a small-scale synthetic setting, the paper finds that ORDERED training data causes facts to be stored in individual slots that are easily updated with little effect on other facts, whereas SHUFFLE training data causes facts that generally move together (e.g. athletes that switch teams together) to be stored together in class-based storage and to generally be updated together. The paper then briefly investigates OLMo-2-7B, a released language model whose pretraining is analogous to SHUFFLE. The authors find that an edit to OLMo-2-7B of one fact moves 9.8% of the other facts sharing its relation and answer and that classes whose answers changed more across pretraining checkpoints are more strongly coupled, hinting at class-based storage which matches the results in the synthetic setting. However, a full analysis is not possible as there is no olmo-equivalent of a model trained on ORDERED data, so this result is correlational and cannot show that the coupling arises from how OLMo observed changing facts, as the authors acknowledge.

Critical strengths

The synthetic setting is comprehensive and tests counterfactual hypotheses to make sure findings are correct. It is well explained and easily understandable. I also think the finding that the order of training data affects how knowledge is stored is novel and interesting, even if it is difficult to thoroughly test and conclude that the same patterns exist in large pre-trained models like OLMo (OLMo is the SHUFFLE equivalent of the synthetic setting but no ORDERED equivalent exists).

Critical weaknesses
  • the OLMo setting is only evaluated on one model; it is unclear whether the results reproduce on models of e.g. different sizes (though these are available; e.g. OLMo 2 is released at 1B, 7B, 13B and 32B).
  • There is only correlational evidence connecting the synthetic and OLMo setting and the experimental setup is not the same in the two settings. The OLMo setting cannot do an actual comparison of ORDERED versus SHUFFLE as the pretrained setting does, as OLMo is the SHUFFLE analogy but no ORDERED analogy exists. This is a problem because the finding that coupling of facts exists in OLMo as SHUFFLE would predict doesn't mean that a model similar to OLMo trained in ORDERED fashion would show facts stored in individual slots, so the analysis is incomplete. Moreover, in the synthetic setting, classes are defined as groups of facts that have changed together throughout training and shows that without this shared history there is no coupling. However, in the OLMo setting, classes are defined as groups of facts that share the same relation and answer without considering if facts changed together during training.
Questions for authors
  • what is representation rank? What is participation ratio? These terms are not defined. It is unclear to me what exactly figure 5a/the respective paragraph in 4.3 is measuring. For example, which activations are used? At which token position? Over what vectors? Is your inference that low rank activations imply that models are not memorizing shifting facts? That in itself is not obvious to me?
  • in an attempt to make the OLMo and synthetic setting more similar, can you repeat the OLMo experiment where you edit facts by finetuning with facts from groups where facts changed together across checkpoints?
2 comments
AuthorsOpenReview ↗

Response to Reviewer

Thank you for your insightful feedback. We were encouraged that they you found our discoveries to be novel and interesting, and the synthetic setting to be comprehensive and well-tested.

Clarifications on Rank

what is representation rank? What is participation ratio? These terms are not defined. It is unclear to me what exactly figure 5a/the respective paragraph in 4.3 is measuring. For example, which activations are used? At which token position? Over what vectors? Is your inference that low rank activations imply that models are not memorizing shifting facts? That in itself is not obvious to me?

We apologize for the lack of clarity in the writing and for omitting an explicit formula of the participation ratio. In all parts of the paper, we measure the representation rank via the participation ratio.

Given a set of sampled activations, whose empirical covariance matrix has eigenvalues λ1,...,λn\lambda_{1},...,\lambda_{n}, the participation ratio is defined as (∑i=1nλi)2∑i=1n(λi)2\frac{(\sum_{i=1}^{n} \lambda_{i})^{2}}{ \sum_{i=1}^{n} (\lambda_{i})^{2}}. This is a measure of how concentrated the variance of the data is.

For example, which activations are used? At which token position? Over what vectors?

In calculating this participation ratio, we take the MLP-input activations of the model directly preceding the position of the answer token. In our main paper results, we used the activations before the final MLP block, but we provide additional results across layers in Appendix J. As in prior formulations of key-value fact storage, this represents the key vector through which the model retrieves its stored answer.

Is your inference that low rank activations imply that models are not memorizing shifting facts? That in itself is not obvious to me?

We do not intend to put forward any inference about whether the model has memorized using these representational results. For clarity, all models that we test here have 95%+ accuracy on the final time-step shifting facts, which suggests that the model does memorize them (regardless of the rank). The role of participation ratio in our analysis is primarily as a metric of how facts are stored in the model internally.

Intuitively, a low participation ratio means that most of the variance across shifting facts lies along a few directions shared by facts in the same class, suggesting less indvidualized representations. A high participation ratio means that each fact also varies along directions of its own, indicating more individualized representations.

OLMo Co-Movement

...in the OLMo setting, classes are defined as groups of facts that share the same relation and answer without considering if facts changed together during training.

...can you repeat the OLMo experiment where you edit facts by finetuning with facts from groups where facts changed together across checkpoints?

We have added a version of this experiment to the revised Appendix K, where we show that updates spread further among facts that changed together in the real world. We consider 24 classes of facts, each group of which changed together during a known hisotrical event. Some examples include a group of AFL teams that joined the NFL together in 1970 or the cities of East Germany in 1990. After an update to part of a group, 71.4% of the remaining members adopted the new answer, compared with 60.2% of classmates that reached the same answer through a different event (+11.1 points, 95% interval [1.5, 19.8]). This analysis, by design, controls for potential confounds of relation type, fact popularity, and difficulty of the edit and supports our findngs of co-movement classes in a pre-trained model.

Thank you for the response and the additional experiments. The clarifications on rank resolve my concern. On OLMo co-movement, my concern is partially addressed: my question concerned groups whose facts changed together across model checkpoints, while the new experiment examines groups whose facts changed together in the real world. This does not establish whether the model learned these facts as changing together during pretraining. Still, the new experiment provides useful evidence that updates spread more readily among facts that changed together historically.

My concern about evaluation on only one pretrained model is unaddressed: it is unclear whether the OLMo results reproduce across model sizes. Another remaining limitation is that there is only correlational evidence connecting the synthetic and OLMo settings, though I appreciate that this is difficult to resolve as no ORDERED-OLMo equivalent exists. I retain my rating of weak accept.

Reviewer_kNmuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper looks into whether models simply memorize each time period separately or learn how knowledge evolves. The authors also investigate whether the order in which these changes are seen shapes how the model absorbs later updates.

To study this, the authors build a controlled synthetic knowledge graph with constant facts and shifting facts that change predictably across time-steps. They train a small transformer from scratch under three presentations: chronological (ORDERED), mixed across time (SHUFFLE), and a single time-step (SNAPSHOT). Each model is then updated on a held-out time-step and probed for stability, plasticity and how far updates spread.

Some important findings: seeing multiple time periods helps the model keep constant facts and learn predictable updates faster. The presentation order determines how facts are stored. And chronological training stores facts in near-independent slots, so updates stay mostly local. However, shuffled training groups facts that changed together into shared "co-movement classes", so an update to one fact spreads to the others. Analyses of embedding geometry, gradient similarity and training dynamics trace this difference to interference between conflicting versions of a fact early in training. In OLMo-2-7B, edits spread mainly among facts that share a relation and an answer, and spread further in groups whose answers changed more across checkpoints.

Critical strengths
  1. The question is under-explored. Most work on knowledge updates studies the edit itself, while this paper asks how seeing facts change during pretraining shapes the way a model responds to later updates, which is a useful angle for continual learning.

  2. The ORDERED vs SHUFFLE contrast is clean and robust. The two conditions see identical facts and differ only in order, and SHUFFLE spreads updates more than ORDERED in every figure. The causal checks also support the mechanism: projecting out SHUFFLE’s shared subspace drops its coupling to ORDERED’s level while a random subspace does nothing (Fig 8), and PCGrad shows that within-batch conflict is not required (App D).

  3. The paper goes beyond reporting the effect and tries to explain it. The phase-length sweep (App F) shows that chronological training couples more as phases shorten, even with plain SGD, which points to interference between conflicting versions of a fact carried by the weights. Random updates (Fig 7), random jumps (App E) and corrupted transitions (Fig 12) also rule out several shortcuts.

4.The writing is pretty clear

Critical weaknesses

There are two things that I want to point out:

  1. The “fast at any future time-step” result could be partly recall. Answers advance on a ring of 32 positions with T=31, so if I understand correctly, the far target are the same configurations the model already saw at τ=1…14 during pretraining. In that case, the paper’s statement that the held-out step “never repeats a step seen during pretraining” only holds for T and T+1. SHUFFLE’s speed at T and T+1 is genuine, but at the far offsets both its speed and ORDERED’s slowness may just reflect remembering or forgetting early time-steps. Restricting the claim to T and T+1, or using a ring longer than T+16, would settle this.

  2. The stability advantage may partly come from fewer update steps rather than from seeing change. Constant-fact retention is measured when the update reaches 95% accuracy, which takes about 110 steps for SNAPSHOT but only about 30 and 10 for ORDERED and SHUFFLE (Fig 2b). Since more gradient steps alone can cause more forgetting, part of the 0.68 vs 0.96 gap might be unrelated to temporal exposure, though I could be wrong here. Because this is claim (a) of the abstract, it would be good to compare retention at matched steps. Fig 7 rules out a similar explanation for plasticity, but not for stability.

Questions for authors
  1. Are the far offsets in Fig 2(b) meant as extrapolation? If targets T+2…T+15 coincide with the τ=1…14 pretraining configurations, could you report each model’s end-of-pretraining accuracy on those answers using the existing checkpoints? If SHUFFLE still recalls them, restricting the “any future time-step” claim to T and T+1 would be enough.

  2. Does the stability gap hold at matched update steps? Could you report constant-fact retention for all three models after the same number of update steps (e.g., 10 and 30)? This only requires evaluating during the short update phase. If the 0.96 vs 0.68 gap still holds, it would resolve my main concern about claim (a).

  3. How were the 947 OLMo edits and 102 classes selected from the 6,136 facts? If it is easy, does ρ=0.28 still hold with relation fixed effects and a class-size control?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1/2

We thank the reviewer for their careful review and are encouraged that they find the question understudied, the results robust, and that the explanation was clear.

Recall versus Extrapolation

The “fast at any future time-step” result could be partly recall. Answers advance on a ring of 32 positions with T=31, so if I understand correctly, the far target are the same configurations the model already saw at τ=1…14 during pretraining. In that case, the paper’s statement that the held-out step “never repeats a step seen during pretraining” only holds for T and T+1. SHUFFLE’s speed at T and T+1 is genuine, but at the far offsets both its speed and ORDERED’s slowness may just reflect remembering or forgetting early time-steps. Restricting the claim to T and T+1, or using a ring longer than T+16, would settle this.

Are the far offsets in Fig 2(b) meant as extrapolation? If targets T+2…T+15 coincide with the τ=1…14 pretraining configurations, could you report each model’s end-of-pretraining accuracy on those answers using the existing checkpoints? If SHUFFLE still recalls them, restricting the “any future time-step” claim to T and T+1 would be enough.

The reviewer's understanding is correct. As the answer rings have 32 positions and pretraining covers 30 time-steps, the targets for τ=T+2,…,T+15\tau=T+2,\dots,T+15 equal each fact’s answer at τ=1,…,14\tau=1,\dots,14 for 90–94% of facts. Only τ=T\tau=T and τ=T+1\tau=T+1 have answers that never appear in pretraining. At the end of pretraining, Shuffle still recalls these earlier answers when queried at their original time-steps (94–99% in the run shown in Figure 2(b)), while Ordered and Snapshot recall none of them. Ordered returns its most recent answer at every time-step. The far offsets therefore do not test extrapolation for Shuffle.

However, the underlying claim remains. On the two targets whose answers are new, Shuffle learns the update within 10 steps for both τ=T\tau=T and τ=T+1\tau=T+1. Ordered takes 30 and 90 steps, and Snapshot takes 110 for both. We will restrict the claim to τ=T\tau=T and τ=T+1\tau=T+1, state in the caption that later targets coincide with answers from pretraining, and report these recall measurements in the appendix.

Stability at Matched Steps

The stability advantage may partly come from fewer update steps rather than from seeing change. Constant-fact retention is measured when the update reaches 95% accuracy, which takes about 110 steps for SNAPSHOT but only about 30 and 10 for ORDERED and SHUFFLE (Fig 2b). Since more gradient steps alone can cause more forgetting, part of the 0.68 vs 0.96 gap might be unrelated to temporal exposure, though I could be wrong here. Because this is claim (a) of the abstract, it would be good to compare retention at matched steps. Fig 7 rules out a similar explanation for plasticity, but not for stability.

Does the stability gap hold at matched update steps? Could you report constant-fact retention for all three models after the same number of update steps (e.g., 10 and 30)? This only requires evaluating during the short update phase. If the 0.96 vs 0.68 gap still holds, it would resolve my main concern about claim (a).

In our revised paper (Appendix H), we show a full update learning curve. Our results demonstrate that across a range of matched step numbers, models exposed to multiple time-steps retain more of the constant facts. For convenience, we also report tabular results below:

Constant-fact accuracy after a matched number of update steps

Update stepsOrderedShuffleSnapshot
0 (before update)1.00 ± 0.001.00 ± 0.001.00 ± 0.00
1000.93 ± 0.050.95 ± 0.020.40 ± 0.02
2000.96 ± 0.030.96 ± 0.010.62 ± 0.06
3000.95 ± 0.040.94 ± 0.010.68 ± 0.00
5000.95 ± 0.020.88 ± 0.010.65 ± 0.02
1,0000.80 ± 0.040.59 ± 0.100.34 ± 0.06
AuthorsOpenReview ↗

Response to Reviewer 2/2

Setup of the OLMO Experiment

How were the 947 OLMo edits and 102 classes selected from the 6,136 facts? If it is easy, does ρ=0.28 still hold with relation fixed effects and a class-size control?

Fact Selection Protocol

We sourced the 6,136 facts from those that the model answered correctly consistenly across surface phrasings (out of 24208 popular subject-relation pairs across 20 relations) present in the data. From this, we sample from groups of facts which lie classes (i.e. are members of groups of entities that have one other ther fact which share the same relation and answer), which produced the final 947 edits.

Selecting edit targets

A class is the set of facts that share a relation and an answer. Edit targets are drawn at random from facts with at least one other member in their class, taking the 20 relations in turn. Each edit installs the relation's most common answer that neither the target nor any fact measured for that edit already gives. We fine-tune all parameters (AdamW, learning rate 10⁻⁵) on two phrasings of the target with the new answer, and stop at the first step at which the model gives the new answer under the third, held-out phrasing. For each edit, we measure up to 20 other members of the target's class and eight facts each that share only the relation, only the answer, or neither. The 947 edits are those that installed, counting each pair of target fact and new answer once.

Relation and Class Size Control

We added new experimental results in Appendix K that explicitly control for relation and class size. We show that updates spread further among facts that changed together in the real world. We consider 24 classes of facts, each containing a group that changed together during a known historical event. Examples include the AFL teams that joined the NFL together in 1970 and the cities of East Germany in 1990. After an update to part of a group, 71.4% of the remaining members adopted the new answer, compared with 60.2% of classmates that reached the same answer through a different event (+11.1 points, 95% interval [1.5, 19.8]).

This comparison controls for relation and class size by construction. The remaining group members and the classmates they are compared with belong to the same class, so they necessarily share the relation, the answer, and the class size. Neither relation nor class size can therefore explain the gap. We provide full details in Appendix K.

Thanks for the response and the new experiments. Most of my critical concerns are resolved. I’m keeping my weak accept since the OLMo evidence is still correlational, as the other reviewers also noted.

Reviewer_LvZyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how models learn from facts that change over time. Through synthetic knowledge graph experiments, the authors find that chronological training leads to more independent fact representations, while shuffled training encourages shared representations among facts that change together. The paper further analyzes these behaviors through gradient and representation studies, and reports similar patterns in OLMo-2-7B.

Critical strengths

The paper investigates an interesting question about how training data order affects knowledge storage and future updates. The controlled experiments provide useful evidence, and the analysis of gradient similarity and representation structure helps explain the observed behavior.

The authors also include several useful experiments on update coupling, temporal noise, and model size. These make the findings in the synthetic setting more convincing.

Critical weaknesses

I have some concerns about novelty. Previous work has already studied fact representations, knowledge editing, and how updates propagate across related facts. The proposed distinction between individual slots and shared classes is interesting, but it is not entirely clear how much new understanding it adds beyond existing observations of shared representations and update coupling.

My main concern is the limited experimental generalization. Most experiments rely on small models and synthetic knowledge graphs with highly structured temporal changes. Although the authors include an OLMo experiment, the reported correlation is relatively weak, and changes in model predictions across checkpoints do not necessarily reflect actual changes in world knowledge. I am not fully convinced that the same mechanisms explain knowledge storage in realistic pretrained models. Also there is no table in the paper only figure which I think is not enough.

Questions for authors

Could the authors clarify the novelty compared with prior work on fact storage and out of context generalization? What makes the two proposed storage mechanisms fundamentally different from previously observed representation sharing?

How would the results change with more realistic temporal patterns, where facts change irregularly and different entities evolve independently? It would also be useful to validate the findings on larger models.

In Figure 2, the models require different numbers of gradient steps to reach the same update accuracy. Could this partly explain the differences in knowledge retention? A comparison under matched update budgets would help.

For the OLMo experiments, could the authors control for factors such as relation type, fact frequency, class size, and editing difficulty? Also, how can changes in model predictions across checkpoints be distinguished from actual temporal knowledge changes?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1/2

We thank the reviewer for their careful review and are encouraged they found the question interesting and experiments useful!

Clarifications on Novelty

...it is not entirely clear how much new understanding it adds beyond existing observations of shared representations and update coupling.

Could the authors clarify the novelty compared with prior work on fact storage and out of context generalization? What makes the two proposed storage mechanisms fundamentally different from previously observed representation sharing?

Prior work treats the geometry of fact storage as determined by the structure of the underlying knowledge graph [1,2], and attributes generalization between facts to the implicit bias of gradient descent [3]. We show that the geometry is not fixed by the facts or their semantic relationships. Ordered and Shuffle see the same knowledge graph with the same model and optimizer, yet training order alone decides whether facts are stored in individual slots or in shared classes. To our knowledge, this impact of training order on fact storage geometry has not been shown in any prior works.

[1] Noroozizadeh, Shahriar, et al. "Deep sequence models tend to memorize geometrically; it is unclear why." arXiv preprint arXiv:2510.26745 (2025).

[2] Nishi, Kento, et al. "Representation shattering in transformers: A synthetic study with knowledge editing." arXiv preprint arXiv:2410.17194 (2024).

[3] Huang, Yixiao, et al. "Generalization or hallucination? understanding out-of-context reasoning in transformers." Advances in Neural Information Processing Systems 38 (2026): 139807-139855.

Structured Changes

How would the results change with more realistic temporal patterns, where facts change irregularly and different entities evolve independently?

We show that exposure to multiple time-steps alters how models respond to updates, even when there is no predictable temporal patterns. In our new experiment (Appendix I), each shifting fact takes a random answer at every time-step, independently of the others (even at the update step), so there is no successor rule and no co-movement. We study whether exposure to multiple time-steps:

  • Improves the retention of constant facts (stability)
  • Reduces the number of steps to absorb the update (plasticity)

In this setting, it does not make sense to measure co-movement, because there is no known structure of facts that change together.

5 seedsOrderedShuffleSnapshot
Stability: constant-fact accuracy when the update reaches 95%0.73 ± 0.080.88 ± 0.030.54 ± 0.06
Plasticity: steps to learn the random update to 95%269 ± 20202 ± 14348 ± 39

We show that that models that are exposed to facts from multiple time-steps (Ordered and Shuffle) learn updates faster and preserve constant facts more than the Snapshot baseline. This suggests that exposure to changing knowledge improves later updates, even on unpredictably changing facts.

Real-World Changes in OLMo

Although the authors include an OLMo experiment, the reported correlation is relatively weak, and changes in model predictions across checkpoints do not necessarily reflect actual changes in world knowledge. I am not fully convinced that the same mechanisms explain knowledge storage in realistic pretrained models.

For the OLMo experiments, could the authors control for factors such as relation type, fact frequency, class size, and editing difficulty? Also, how can changes in model predictions across checkpoints be distinguished from actual temporal knowledge changes?

We address all of these points with a new experiment on OLMo-2-7B (Appendix K), where we show that updates spread further among facts that changed together in the real world. We consider 24 classes of facts, each group of which changed together during a known hisotrical event. Some examples include a group of AFL teams that joined the NFL together in 1970 or the cities of East Germany in 1990. After an update to part of a group, 71.4% of the remaining members adopted the new answer, compared with 60.2% of classmates that reached the same answer through a different event (+11.1 points, 95% interval [1.5, 19.8]). We provide full details in Appendix K.

This experimental design addresses each of the desired controls:

  • Relation type and class size. Both sets belong to the class of the updated facts, so they share the relation, prompt templates, answer before the update, class size and new answer. A shared answer therefore cannot explain the difference.
  • Fact frequency and editing difficulty. Both sets are measured after the same update, so the updated facts, their popularity, and the steps needed to install the update are identical.
  • Model predictions vs. actual changes. The groups come from documented events, so they reflect actual changes in the world rather than changes in the model's predictions.
AuthorsOpenReview ↗

Response to Reviewer 2/2

Controlling for Update Steps

Could this partly explain the differences in knowledge retention?

We find that it does not. The stability gap holds at every matched budget, so it is not a side effect of update speed. From 100 to 1,000 update steps, Ordered and Shuffle have 0.23 to 0.55 higher constant-fact accuracy than Snapshot (curves in new Appendix H).

Constant-fact accuracy at matched update steps

Model30 steps100 steps200 steps
Ordered0.98 ± 0.010.99 ± 0.010.99 ± 0.01
Shuffle0.99 ± 0.000.99 ± 0.010.99 ± 0.01
Snapshot0.42 ± 0.000.62 ± 0.020.55 ± 0.02

Dear Reviewer, this is the area chair for this paper. The authors have provided extensive responses to your review comments. I hope you can take some time to go through the responses and offer your final remarks. Many thanks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.