The Linguistic Long Tail Reshapes Language Models

The linguistic long tail contains languages scarce in data yet rich in linguistic structures underrepresented during foundation-model pretraining. Continued pretraining (CPT) is typically used to expand what foundation models know; we show that learning new linguistic structure can instead reshape behaviors they already possess. CPT on small corpora of polysynthetic languages induces 31.2% shorter English generation despite containing no English adaptation data, while preserving lexical fact coverage; matched analytic-language and English CPT do not reproduce the effect. The shift extends to mathematical reasoning and generalizes across model scales. Causal interventions localize most of the effect to CPT-induced updates in a band of intermediate layers; scaling these updates provides graded control over generation length across tasks. These findings position the linguistic long tail as a scientific instrument for understanding foundation models: long-tail languages can reshape broader model behavior.