Lasting Effects of Abstract Pretraining Beyond Perplexity

Language models are typically pretrained from randomly initialized weights. Recent work challenges this convention, showing that a brief warm-up on abstract, algorithmically generated data can provide a useful starting point for subsequent learning from natural language. In this paper, we show that, in small language models, such a warm-up improves the acquisition of downstream capabilities in ways that are not reflected in language-modeling perplexity. Our warm-up uses an abstract stack-manipulation task in which models process sequences of push and pop operations and predict the remaining contents. Although it uses only approximately 1% as many tokens as the language-pretraining phase that follows, it improves multi-hop question answering after fine-tuning by up to 3.9 F1 points on MuSiQue, with additional gains on HotpotQA and 2WikiMultihopQA, despite comparable language-modeling perplexity. Controlled experiments further show that the warm-up substantially accelerates the acquisition of deeper reasoning chains. We then explore what drives this transfer. First, it depends on the computation learned: replacing the stack's last-in-first-out rule with a matched first-in-first-out rule eliminates the downstream advantage. Second, transfer is selective: gains concentrate on dependent reasoning chains, with no consistent benefit for tasks that merge or compare independent lookups. Third, timing matters: exposure before language pretraining produces the largest gains, during pretraining yields smaller gains, and afterwards has no improvement. Moreover, the early advantage persists through billions of subsequent language tokens. Together, these results show that early training on abstract algorithmic data can shape the capabilities that language models later acquire, producing benefits that language-modeling perplexity does not capture.