acceptodds
Under review as a conference paper at ICLR 2027

Long-Tailed Entity Distributions Preserve World Model Plasticity

Abstract

Our world is ever-changing and, while language models can incorporate new information flexibly in-context, they face difficulties doing so *in weights*, in a way that generalizes. We show that the pre-training data distribution is a key lever: long-tailed entity distributions preserve a model's ability to integrate new entities into its world model. We study this in a controllable synthetic world where cities with known geometry underlie multiple relational tasks. We first show that models use the representational geometry of entities when solving tasks, and that how well a new entity is integrated into the world geometry predicts its cross-task generalization (). We then find that under uniform pre-training, the world representation forms early but loses plasticity: its ability to integrate new entities via gradient descent decays with continued training, even as pre-training loss decreases. Pre-training with Zipfian entity frequencies counters this, improving new-entity integration on average and on the hardest task, and outperforming 15 alternatives including meta-learning. We hypothesize that sparse exposure keeps some entities effectively new (i.e. incompletely learned), providing continued practice integrating them throughout pre-training. At high Zipf , however, tail learning and world-model quality deteriorate. In all, we establish that long-tailed entity distributions help maintain the plasticity of entity representations and reveal a trade-off between forming a world model and keeping it open to change.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, world modeling, fine-tuning, representation learning, generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9SqWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study the representations models learn when trained on synthetic location data: concretely, they train LMs to solve tasks related to the distance angle between, angle between, etc. different cities. The use the same setup as previous work (Park '26) and extend it in two ways: 1) they intervene on hidden states for different locations, finding that displacing and interpolating the hidden representations gives the expected geographic readout 2) they find that training on new locations (e.g. fictional "Atlantis" cities) performs better if the intial pretraining occurs with long-tailed distributions (sampling from a Zipf distribution) compared to uniform sampling, implying that maintaining representational plasticity through pretraining on long-tailed data facilitates later finetuning.

Critical strengths
  • Understanding the dynamics of how and when learned representations generalize is an important and open question. This is a clever synthetic setting and the trade-off between plasticity and structure is insightful.
  • The Zipf distribution is empirically observed in natural language (and in many domains), so it seems plausible that the good results we see for autoregressive models across a variety of domains is related to balancing this plasticity and structure tradeoff.
Critical weaknesses
  • My main concern is that this paper seems not fully baked. The contributions are incremental to Park '26 (that shares the same setup) and large sections smell of LLM writing (difficult to interpret results dumps lacking clarity or context). While I think this is interesting and promising work, this needs quite a bit of polishing and potentially some additional contributions.

Clarity

  • Is the normalized improvement metric provided anywhere other than the Park '26 reference?
  • Figure 8: It would be good to give a bit more context in the caption; I had to backtrack quite a bit through the paper to understand what is happening here
    • In fact, most of the captions could use a bit more context.
Questions for authors
  • In the interpolation, etc. experiments: Is there any interpretable notion of hidden state distance that corresponds to output space distance?
  • Is there anything noticably different in the representations between the Zipf training vs the uniform training that could lead to the observered difference in plasticity? Do you see anything interesting in the new representations that you are trying to train in when they "work" vs "don't work"?
5 comments

Thank you for your thoughtful review! We were glad to hear that you found our question important and the trade-off we identify insightful. We’d be happy to address your concerns below.

Needs quite a bit of polishing

We appreciate your feedback about the clarity of our writing. We recognize that parts of our paper need more context and clarity, and will continue revising our text. We kept the figure captions very brief to meet the page limit and relied on the surrounding text to explain our findings. Now that we have an additional page, we will extend our captions and make our figures more self-contained.

Large sections smell of LLM writing

We used some AI assistance in our writing process, but our paper’s main body is substantially human written.

Contributions incremental to Park ‘26

We do share Park (2026)’s setup, but our work’s central question is about how pre-training shapes a model’s subsequent plasticity (i.e. ability to learn new entities). We do so via 3 contributions: (1) showing how world model representations are used in model computations and how newly learned representations can be reused by existing computations; (2) showing how plasticity declines during pre-training and identifying how pre-training data distributions remedy this; and (3) examining which aspects of the proposed distributions contribute to these benefits and their trade-offs. We feel these goals and investigations are meaningfully different from Park’s.

Is the normalized improvement metric provided anywhere?

We describe normalized improvement in App. B.3 (L1020-1037). We will add a brief definition into the main text.

Is there any interpretable notion of hidden state distance that corresponds to output space distance?

Yes. In the uniform pre-trained model, Euclidean distances between real-city representations at the intervention layer strongly correlate with the model’s generated distance answers (r2=0.91±0.02r^2 = 0.91 \pm 0.02, mean ±\pm sample SD across three seeds).

Differences in representations

We understand that the real-city representations are substantially similar after uniform and Zipfian pre-training. In both cases, models learn world-aligned representations and all pre-training tasks operate on these world-aligned representations (e.g. rather than learning city representations independently per task).

There is a difference in new city (Atlantis) representations between settings. When Atlantis cities are integrated in the world model (e.g. via Zipfian pre-training), these Atlantis representations can be reused by the task circuits learned from pre-training, which enables cross-task generalization. Otherwise (e.g. with uniform pre-training and distance fine-tuning), the model effectively learns Atlantis representations independently from the existing world model, and produces a solution specialized to the fine-tuning examples. This solution generalizes poorly to other tasks and to queries between two Atlantis cities (which is underrepresented in the fine-tuning data; rather than between an Atlantis city and real city). This is consistent with reduced reuse of the existing world model.

===

We hope to have addressed your concerns! Please consider raising the score if so. Please let us know if you have additional concerns or questions.

Thanks for the response. I'm not sure I understand your last point, though. My questions about the representations had more to do with the geometry of the representation space: is the structure of the Zipfian training representations somehow more interpretable in a way that shows clear integration of new entities (where the uniform pretraining does not)?

While the work is interesting, I still think this paper is not fully developed and would need either several more experimental settings or an additional contribution to be merit acceptance.

Putting it simply, the geometry of representations is mostly similar between the pre-trained models (uniform vs zipfian). You can see this in the decoded real-city representations visualized in gray in Fig. 6. You'll notice that the cities are slightly more diffuse in the zipfian (alpha = 1.25) model. [Note: Fig. 6 is post-fine-tuning, but the gray cities look the same as post-pre-training.] Fig. 5(a) shows how this harm to the real-city world model is a trade-off w.r.t. plasticity, although our results don't establish that it is the cause for plasticity.

Our feeling is that our contributions (listed 1-4 in the Introduction, L80-100) are thorough and meaningfully independent from what is investigated in Park (2026). Are any of these listed contributions under-developed? Or could you please say more substantively what kinds of additional contributions are missing?

Reviewer_zqwLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether the distribution of the pre-training data influences how well a model is able to learn new entities during fine-tuning. For this purpose it makes use of the synthetic city world, in which small Transformers are first pre-trained on seven geometric tasks using 5,075 real cities before being fine-tuned on one task with 100 new "Atlantis" cities. The authors first demonstrate that city representations are involved in the computational process and that the reconstruction error for the Atlantis probe correlates with cross-task generalisation (r2r^2 = 0.93). When entities are sampled uniformly, plasticity decreases as pre-training progresses. Sampling according to a Zipf distribution with α\alpha = 1.25 reduces the Atlantis reconstruction error by about a factor of three (from 409 to 133) and outperforms 15 other interventions, while staging and scarcity controls are provided as evidence that the advantage arises because some entities are kept 'effectively new'.

Critical strengths
  1. That is a reasonable question and the way it's set up works because in a situation where the target geometry is known, it is possible to measure integration directly by asking whether updatability is determined during pre-training, something that natural text cannot do.

  2. Certain controls are very well designed; with regard to distance, the random-row release using the same pool-size schedule achieves 852 compared to 197 for staging (see Fig. 11a), which essentially eliminates 'fewer rows early' as an explanation. The tail-replay reversal (from 335 to 878) provides a nice test of the scarcity hypothesis. The geometry interventions in Section 3 include both equal-norm random edits and an alternative based on endpoint arithmetic, something that I liked.

  3. There are three seeds with SD values given in every case, and the appendix is open about its limitations. Appendix F refers to the alternatives as 'selected settings, not equal-compute family rankings', while Appendix H points out the replay asymmetry. The findings regarding relocation and the held-out task (Figure 8) are a valuable extension.

Critical weaknesses
  1. My primary worry is that the 3× headline compares models which have very different world models, since the paper's own results indicate that a large part of the difference is simply due to slower learning. When α is 1.25, the existing world is learned much worse: the real-city reconstruction error reaches 46 compared with 16 in the uniform case (Section 5). In Figure 17 the real-city distance MAE rises from about 4 to somewhere between 30 and 40, and the triangle area and angle are also several times worse.

Fig. 9 indicates that uniform checkpoints are quite plastic in the early stages. When pre-training reaches 15% the seven-task Atlantis error for uniform is about 110, this being less than Zipf's final figure of approximately 155 on the same test. At 20% the real-city error for uniform (48) is the same as Zipf's final real-city error (46), and its seven-task Atlantis error is about 175. Thus, when the geometry is matched the difference is small. Distance, however, is the exception: uniform's distance error rises from about 170 at 15% to about 560 at 20%, whereas Zipf's is close to 235.

I'm not certain that Zipf "preserves plasticity" rather than shifting the model to a different point on the trade-off curve which the paper itself mentions in contribution 4. What would truly support the title is a better frontier, that is to say achieving a lower Atlantis error while maintaining the same level of performance on the real-city task, and the checkpoints required to plot this frontier already exist. As for this point, the "Checkpoint / LR" family in Fig. 7 only considers an 80% checkpoint, even though Fig. 4(b) clearly shows that early stopping should be the obvious alternative to compare it with.

  1. The benefits from the representation aspect are much greater than those from behaviour, which is why the paper highlights the representation figure. The 3× refers to the probe reconstruction error. In the case of cross-task NI, Figure 5(e) shows the seven-task average to be about 0.57 when α = 1.25 as compared to 0.437 for the uniform case; that average is my own estimation from the bars, whereas the main text provides this value for staging and scarcity but not for Zipf. For angle and perimeter, the reconstruction error is reduced by more than half while the NI value decreases slightly (from roughly 0.74 to 0.71 and from 0.71 to 0.66).

This is important since the argument in favour of taking reconstruction as the outcome is based on r² = 0.93 in Fig. 3(b). The fit referred to here is based on seven task means obtained under uniform pre-training, and the distance is far out at approximately (850, 0.08) as a high-leverage point. The fact that there is a correlation between the fine-tuning tasks does not imply that an intervention which reduces reconstruction error will also improve generalisation by an equivalent amount, and Fig. 5 demonstrates that this is often not the case.

  1. The comparison with the 15 alternatives is not on an equal basis. The value of α was varied over 41 values (from 0 to 2 in steps of 0.05) and was selected at the point where the Atlantis error reaches its lowest level in Fig. 5(a). This indicates that the choice of α makes use of the evaluation entities themselves and does not involve a separate group of new entities for selecting α. Most of the alternatives are given only one set of parameters (dropout at 0.1, weight decay at 0.1, character noise at 0.05), MAML receives 327 meta-updates per task, and App. F acknowledges that these are not equal-compute comparisons. The statement in the abstract that it "outperforms 15 alternatives including meta-learning" amounts to a stronger claim than the evidence actually supports. In my view, α should be chosen using a held-out set of new entities, or there should be comparable sweeps carried out for at least the two or three best alternatives.

  2. Although the mechanism—"keeping entities effectively new"—is plausible, it has only been tested indirectly. Both staging and scarcity affect a number of factors at the same time, a point that App. H admits. The main Zipf experiment also makes fine-tuning adjustments using uniform real-city replay (App. B.2). As a result, the fine-tuning mixture of the Zipf model includes a large number of tail cities that it knows very little about, and some of the "practice" might in fact be occurring during fine-tuning rather than during pre-training. Since scarcity still provides a benefit (154), this partly resolves the issue. That said, the main Zipf result should be rerun using Zipf-distributed replay. It would also be useful to carry out a direct test, for instance by checking how much the representations of tail entities are still changing late in pre-training and whether that change predicts plasticity across α.

  3. Regarding scale, the system makes use of a single 1.6M-parameter model (with six layers and a width of 128), operating at the character level in a 2D world containing 5,075 entities. The collapse that occurs at high α resembles a capacity limitation in which the tail elements are underlearned. Since we have only one width, we are unable to determine whether the loss of plasticity or the optimal α changes as the model size increases.

Questions for authors
  1. This one decides my score. Can you plot Atlantis reconstruction error and cross-task NI against real-city task performance, over all pre-training checkpoints, for α in {0, 1, 1.25}?

  2. What's the seven-task average cross-task NI for α = 1.25? What's r² in Fig. 3(b) without the distance task, or with seed-level points?

  3. Was α = 1.25 chosen by looking at Atlantis error? If you selected α on a separate set of synthetic new entities (the 64 used for the sharpness penalty, say), would it land somewhere similar?

  4. Lower priority, and it won't decide my score: does the Zipf benefit hold up with Zipf-distributed fine-tuning replay, and at one larger model width?

1 comment

Thank you for your thorough review! We were glad to hear that you found our question and experiments well designed. We’d be happy to address your concerns below.

W1/Q1: When α\alpha is 1.25, the existing world is learned much worse

As requested, we have plotted reconstruction error and cross-task NI against real-city task performance over all checkpoints, and have included these (for the distance fine-tuning task) in the supplementary materials. Our paper is up front about the trade-off between retaining the real city world model and plasticity, which is more clearly shown here. Earlier checkpoints indeed offer another axis for interpolating this trade-off. Our plots show that α=1.0\alpha = 1.0 or 1.251.25 models achieve lower Atlantis reconstruction error and higher cross-task NI than evaluated uniform checkpoints with at least as good real-city task performance. The degradation in the real city world model for α=1.25\alpha = 1.25 may be acceptable, given the difficulty of instilling plasticity at all via many other interventions (Fig. 7). However, if one prefers to further preserve real city task performance, the 80% checkpoint for α=1.0\alpha = 1.0 may be more suitable.

W2/Q2: Paper highlights improvements to representations, which are greater than benefits to behavior

We highlight improvements to representations, because it is fundamentally desirable for neural networks to implement models that are aligned with the structure of the real world. We aren’t making the claim that improvements to reconstruction error improve generalization by an equivalent amount. But we do demonstrate a very strong correlation between representational integration and cross-task NI, and our results imply that better integration helps to a certain extent. I.e. in Park (2026), the fine-tuning tasks, besides distance, already resulted in well-enough integrated Atlantis representations and so we don’t expect to see meaningful gains to cross-task NI in those cases.

As requested: For α=1.25\alpha = 1.25, the average cross-task NI is 0.57 (across all fine-tuning tasks). In Fig. 3(b), r2=0.93r^2 = 0.93 with distance and 0.970.97 without. Using seed-level points, r2=0.90r^2 = 0.90 with distance and 0.890.89 without.

W3/Q3: Selecting α\alpha on a separate set of synthetic new entities

Indeed, α=1.25\alpha = 1.25 was chosen by looking at Atlantis error and also considering the tradeoff with degradation to the real city world model (Fig. 5a). We now tried adding another 100 Atlantis cities as a validation set, compared α\alpha values in {0, 0.5, 0.75, 1.0, 1.25, 1.5, 2.0}, and found that α=1.25\alpha = 1.25 continues to be optimal if choosing based on validation set reconstruction error.

Although we did not assign equal compute to the compared methods (some more, some less), we used some reasonable defaults for these well-established hyper-parameters. We conducted some more ablations (covering 27 combinations of LR schedules) in our response to [oBt7]. Zipfian pre-training lowered reconstruction error in 26 of 27 distance-fine-tuning settings and increased cross-task NI in all 27.

W4/Q4: Zipf-distributed fine-tuning replay

We can also use a Zipfian distribution (α=1.25\alpha = 1.25) for the real-city replay fine-tuning data. On average over fine-tuning tasks, we continue to see a 72% reduction in reconstruction error and a 27% gain in cross-task NI when using the Zipfian (α=1.25\alpha = 1.25) pre-training distribution.

W4: How much representations of tail entities are still changing late in pre-training?

We also tracked representations of long-tail entities during the second half of pre-training, correcting for global shifts and rotations. Across seven values of α\alpha and three seeds, larger representation changes were associated with lower new-entity reconstruction error after fine-tuning (partial r=−0.48r = -0.48) and higher cross-task NI (partial r=0.36r = 0.36). These correlations control for final real-city reconstruction error, real-city task performance, and seed.

W5/Q4: Model size

We can also vary the hidden size (128, 256, 512; at 1.6M, 6.4M, 25M parameters) and corresponding attention heads or layers (6, 8, 10, 12; with 1-3M parameters). On average, reconstruction error decreases from 381±34381 \pm 34 (mean ±\pm sample SD) for α=0\alpha = 0 to 184±45184 \pm 45 for α=1.25\alpha = 1.25. Cross-task NI respectively increases from 0.43±0.020.43 \pm 0.02 to 0.49±0.090.49 \pm 0.09. We sweep α\alpha in 0,0.8,1.0,1.2,1.25,1.4,1.5{0, 0.8, 1.0, 1.2, 1.25, 1.4, 1.5} and find that 1.25 continues to be in the optimal range.

===

Altogether, we have shown that Zipfian pre-training distributions yield better checkpoints with respect to the trade-off than uniform distributions, we clarify the significance and expectations of representational integration, and provide additional ablations to support our findings.

We hope to have addressed your concerns! Please consider raising the score if so. Please let us know if you have additional concerns or questions.

Reviewer_oBt7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This study finds that pre-training data distribution impacts a model's ability to update its world model. The authors find that in a synthetic toy setting, long-tailed pre-training data distributions help maintain plasticity better than a uniform distribution.

Critical strengths

The writing is very concise/clear, and the problem is well motivated. The experiments are thoughtfully done, and the toy setting is also a tractable and simple way to (initially) study this problem.

Critical weaknesses

This setting is very toy; the model sizes are quite small and the data is also fully synthetic and doesn't reflect the structure of real world data. Therefore, it's unclear that there results have any implications in a real-world LLM setting (relevant since it appears this paper is motivated by "language models incorporating new information via weights"). Could be interesting to try to setup an experiment thats somewhat less toy and closer to real-world regime and see if findings still hold.

Questions for authors
  1. Have there been ablations done on the optimal learning rate/scheduler for uniform vs zipf pre-training distributions? Right now both use the same but it's unclear if both distributions are optimal under the same hyperparameters.
  2. What happens as you scale model size?
  3. Would these results hold in a less toy setting? How could an experiment be setup to do that?
1 comment

Thank you for your thoughtful review! We were glad to hear that you found our presentation clear, and problem and experiments thoughtful. We’d be happy to address your concerns below.


Q3: Would these results hold in a less toy setting?

We agree that it is also important to evaluate more realistic settings, but feel that this would be more suitable for subsequent work. Our setting is intentionally simple so that we can conduct highly controlled experiments, and still resembles elements of pre-training (e.g. multi-task sequence modeling with Transformers, where the data generating process is disentangled from the underlying world structure). In this work, we aim to investigate the more fundamental topic of how neural networks learn representations (rather than actual language models). We feel that a greater scope would introduce many more design decisions and would require more space than available to properly investigate. We should also note that there is much established precedence for ICLR papers to operate in fully-synthetic domains (see e.g. [1, 2]) and Park (2026) was itself published at ICML 2026.

Q2: What happens as you scale model size?

Our model size reflects one that is sufficient for learning the task at hand. We can also vary the hidden size (128, 256, 512; at 1.6M, 6.4M, 25M parameters) and corresponding attention heads or layers (6, 8, 10, 12; with 1-3M parameters). On average, reconstruction error decreases from 381±34381 \pm 34 (mean ±\pm sample SD) for α=0\alpha = 0 to 184±45184 \pm 45 for α=1.25\alpha = 1.25. Cross-task NI respectively increases from 0.43±0.020.43 \pm 0.02 to 0.49±0.090.49 \pm 0.09.

Q1: Ablations on optimal learning rate and scheduler

We can ablate 27 combinations of pre-training LR (1e-4, 3e-4, 1e-3), pre-training scheduler (linear, constant, WSD), and fine-tuning LR (3e-6, 1e-5, 3e-5; with linear decay). For the distance FT task, uniform pre-training results in 756±225756 \pm 225 average Atlantis reconstruction error (mean ±\pm sample SD) whereas Zipf (α=1.25\alpha = 1.25) results in 71% less, 222±87222 \pm 87. Zipfian pre-training yields lower reconstruction error in 26 of the 27 distance-fine-tuning settings and higher cross-task NI (on average: 0.42 vs 0.17) in all 27. There is one exception (pre-training with WSD at LR 1e-3, fine-tuning at LR 3e-5) where uniform distribution results in lower error (152) than Zipfian (224), although cross-task NI is still 0.04 lower than Zipfian.

===

Altogether, we feel that the current task setup and experimental scope are suitable for our primary research question, and we have provided additional ablations that continue to demonstrate our findings in other settings.

We hope to have addressed your concerns! Please consider raising the score if so. Please let us know if you have additional concerns or questions.

References:

  1. Li, ICLR 2023. “Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task”.
  2. Akyürek, ICLR 2023. “What learning algorithm is in-context learning? Investigations with linear models”.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.