SOURCE-LINKED INTELLIGENCE

Global Divergence, Local Convergence: Representation Geometry in SSMs and Transformers

arXiv · AI, language, vision and robotics · article · Sep 8, 2026 · UTC

Recent state-space models (SSMs) such as Mamba achieve language modeling performance comparable to transformers despite relying on fundamentally different architectures. This raises an important question: how do these structural differences influence the geometry and functional nature of their internal representations? We study this question through a multi-scale analysis of representations in transformers, SSMs, and hybrid architecture. First, we find that SSMs distribute their representational information evenly across all dimensions, whereas transformer representations are heavily dominated

Read original source ↗ Open in workspace

recordType

paper

region

Global