GRAFT: Growing Missing Knowledge through Topology-Guided Data Selection

Continual pretraining requires selecting data that expands a language model's knowledge without eroding what it has already learned. We introduce a method for constructing latent knowledge as layer-wise directed graphs of concept connections. Building on this construction, we develop GRAFT, a data-selection framework that prioritizes documents with weak concept connectivity in middle layers. Training on the selected data strengthens links among previously isolated concepts and improves downstream performance. Across ten knowledge and reasoning benchmarks and three base models, GRAFT achieves the best overall average rank among the evaluated data-selection methods. In our experiments, GRAFT also achieves higher scoring throughput on long documents than the evaluated model-adaptive baselines. When selecting from a mixture of mathematics and STEM-specialized data, GRAFT improves both general knowledge and mathematical reasoning. Finally, we examine how latent knowledge structures in the middle layers of LLMs relate to reasoning and forgetting: improved reasoning coincides with new connected concepts, whereas forgetting is associated with the loss of established ones.