SOURCE-LINKED INTELLIGENCE

Disentangling Representation Evolution in Transformers through Directional Decomposition

arXiv · AI, language, vision and robotics · article · Sep 14, 2026 · UTC

Transformer representations evolve through learned additive transformations that either preserve their current direction or redirect it. We study this evolution as a functional geometry, decomposing learned updates into parallel and perpendicular components. Across pretrained models, we find substantial parallel components beyond the residual identity path. We then apply the decomposition in two spaces: to attention and MLP updates relative to the hidden state, and to attention value aggregation relative to the current token's value. Targeted edits reveal a strongly space-dependent asymmetry:

Read original source ↗ Open in workspace

recordType

paper

region

Global