Graph Machine: Pretraining with Sparse Dynamic Pointers

We introduce the Graph Machine (GM), an architecture that maintains an $O(n)$-sized state and accesses it through sparse, dynamic routing. Unlike methods with fixed-size states or sparse but static routing, GM preserves $O(n)$ complexity in its sparse layers without restricting the potentially accessible state size to $O(1)$. Instead, GM uses pointer-like edges with discrete indices and differentiable weights, updated through a referral mechanism resembling pointer chasing. We replace 75% of Qwen3-0.6B's dense Transformer layers with GM sparse layers and train the resulting models end to end from scratch on 15.7B tokens using a standard pretraining recipe. With only 2 of 4,096 tokens retrieved per KV head in each sparse layer, loss degrades only slightly; with 4, the best model marginally improves loss.