SOURCE-LINKED INTELLIGENCE

Training Trajectories Determine Circuit Removability in Annealable Soft-Prior Transformers

arXiv · AI, language, vision and robotics · article · Sep 9, 2026 · UTC

Soft positional priors can help small Transformers learn retrieval circuits, but it is unclear whether the resulting circuits remain functional once the prior is removed. We test this with an annealable soft-prior Transformer whose attention biases can be learned, faded, or zeroed during training and evaluation. On associative recall, unforced models perform well with the prior active ($0.772 \pm 0.020$) but collapse at zero gate ($0.095 \pm 0.009$). Smooth fade-to-zero training preserves high zero-gate accuracy ($0.734 \pm 0.028$), whereas forced-zero training, hard switching, and post hoc co

Read original source ↗ Open in workspace

recordType

paper

region

Global