SOURCE-LINKED INTELLIGENCE

High-Dimensional Learning Dynamics of Attention-Indexed Models

arXiv · AI, language, vision and robotics · article · Sep 3, 2026 · UTC

Attention mechanisms are central to modern foundation models, yet their training dynamics remain poorly understood, especially when the attention matrices have extensive rank. In this work, we study attention-indexed models, a broad framework that can represent multi-layer and multi-head attention architectures. First, we show that, in a suitable high-dimensional limit, the population-loss landscape is characterized by a finite set of trace order parameters. In contrast, online stochastic gradient descent (SGD) is governed by an infinite hierarchy of matrix moments, which we show can be expone

Read original source ↗ Open in workspace

recordType

paper

region

Global