SOURCE-LINKED INTELLIGENCE

No Equivariant Architecture Covers All Equivariant Attention

arXiv · AI, language, vision and robotics · article · Aug 31, 2026 · UTC

We give a complete characterization of equivariant multi-head self-attention (MHSA): if an MHSA layer is equivariant to a symmetry group $G$, then $G$ can only act by permuting head-clusters, with QK and OV matrices satisfying an equivariance constraint tied to the group action. As a consequence, we prove that any fixed MHSA architecture that achieves exact equivariance by polynomially parameterizing unconstrained MHSA parameters inevitably leads to expressivity loss within the class of equivariant maps: the equivariance locus of unconstrained MHSA forms a union of extremely many Zariski-irred

Read original source ↗ Open in workspace

recordType

paper

region

Global