SOURCE-LINKED INTELLIGENCE

Musec: MomentUm SpEctral Clipping for Stable Muon-type Training

arXiv · AI, language, vision and robotics · article · Sep 10, 2026 · UTC

Muon has emerged as a highly effective optimizer for large language model training, often achieving superior convergence and performance compared with the widely adopted Adam and AdamW optimizers. Nevertheless, Muon is prone to training instability due to its spectral flattening, manifested by loss spikes and unbounded growth of model weights. Existing approaches primarily rely on weight or attention-logit clipping, which require architecture-specific modifications and do not directly address instability across all model components. We propose MomentUm SpEctral Clipping (Musec), which replaces

Read original source ↗ Open in workspace

recordType

paper

region

Global