Long-Tailed Entity Distributions Preserve World Model Plasticity

Our world is ever-changing and, while language models can incorporate new information flexibly in-context, they face difficulties doing so *in weights*, in a way that generalizes. We show that the pre-training data distribution is a key lever: long-tailed entity distributions preserve a model's ability to integrate new entities into its world model. We study this in a controllable synthetic world where cities with known geometry underlie multiple relational tasks. We first show that models use the representational geometry of entities when solving tasks, and that how well a new entity is integrated into the world geometry predicts its cross-task generalization ($r^2=0.93$). We then find that under uniform pre-training, the world representation forms early but loses plasticity: its ability to integrate new entities via gradient descent decays with continued training, even as pre-training loss decreases. Pre-training with Zipfian entity frequencies counters this, improving new-entity integration $3\times$ on average and $4.9\times$ on the hardest task, and outperforming 15 alternatives including meta-learning. We hypothesize that sparse exposure keeps some entities effectively new (i.e. incompletely learned), providing continued practice integrating them throughout pre-training. At high Zipf $\alpha$, however, tail learning and world-model quality deteriorate. In all, we establish that long-tailed entity distributions help maintain the plasticity of entity representations and reveal a trade-off between forming a world model and keeping it open to change.