SOURCE-LINKED INTELLIGENCE

Attention Mean Fields Predict Average Representation Dynamics and Reveal Context-Specific Computation

arXiv · AI, language, vision and robotics · article · Sep 14, 2026 · UTC

A language model's representation geometry is not predetermined; it evolves as the model runs. A faithful account of that geometry must capture that dynamic process, and so cannot be based solely on model-independent statistics such as co-occurrence. Here we introduce a mean-field analysis of attention. The average attention from one token to another defines a kernel that carries representations layer to layer and can be iterated through the network to model how the geometry is transformed. We condition this average two ways. Conditioned on a whole corpus, the kernel predicts the average-case

Read original source ↗ Open in workspace

recordType

paper

region

Global