SOURCE-LINKED INTELLIGENCE

Graph Machine: Towards Better Pretraining via Edges

arXiv · AI, language, vision and robotics · article · Sep 2, 2026 · UTC

We introduce the Graph Machine (GM), an architecture that maintains an $O(n)$-sized state and accesses it through sparse, dynamic routing. Unlike methods with fixed-size states or sparse but static routing, GM preserves $O(n)$ complexity in its sparse layers without restricting the potentially accessible state size to $O(1)$. Instead, GM uses edges - pointer-like objects updated differentiably by a referral mechanism resembling pointer chasing. We replace 75% of the dense Transformer layers in Qwen3-0.6B with GM sparse layers and pretrain from scratch on 15.7B tokens. With only 2 of 4,096 toke

Read original source ↗ Open in workspace

recordType

paper

region

Global