Pre-carved Niches: The Formation Dynamics of Modular Task Partitions in Early LLM Training

Large language models exhibit task-related modular organization. To explain why such organization is already visible before training, we examine its origins and subsequent evolution across multiple model families and parameter scales, using tasks spanning language, mathematics, and physics. Neuron groups are identified by their estimated contributions to task predictions, and their patterns of sharing and separation are tracked during training. Reproducible patterns are present before any parameter update. Controlled experiments further show that changing input structure or prediction targets alters the associated neuron groups and their overlap while model parameters remain fixed. Task characteristics thus already shape how neurons contribute to predictions in an untrained network, producing structured patterns of sharing and separation across tasks. During subsequent training, these relationships evolve in ways that differ across models. These findings establish the role of task characteristics in initial task organization and explain why modular structure can be visible before learning begins.