SOURCE-LINKED INTELLIGENCE

SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

arXiv · AI, language, vision and robotics · article · Sep 1, 2026 · UTC

Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size, conflating architectural advantage with extra FLOPs. We study looping on Mixture-of-Experts Transformers while closely matching per-token FLOPs, total non-embedding parameters, and KV cache. Through a series of ablations, we arrive at a recipe we call SMELT (Sparse MoE Transformer, middle layers Loop Twice), which loops the middle half of layers twice while matching the unlooped Baseline on all three budgets. We scale SMELT across four sizes up to 54B non-embedd

Read original source ↗ Open in workspace

recordType

paper

region

Global