SOURCE-LINKED INTELLIGENCE

A Unifying Perspective on Language Model Representations: From Filler-Role Structure to Mechanistic Interpretability

arXiv · AI, language, vision and robotics · article · Aug 29, 2026 · UTC

A wide range of methods have been proposed for interpreting language models, delivering important insights into their inner workings. However, different methods and their resulting insights stand in relative isolation: what could the underlying structure of language models be, such that they give rise to all our interpretations? In this work, we propose using Tensor Product Representations (TPRs) as a unifying hypothesis. TPRs give a concrete proposal for how compositional structure could be represented in vector space --- as filler-role bindings. We show, both mathematically and empirically,

Read original source ↗ Open in workspace

recordType

paper

region

Global