SOURCE-LINKED INTELLIGENCE

Blog: Survey of Optimizers

arXiv · AI, language, vision and robotics · article · Aug 28, 2026 · UTC

Neural-network optimization in 2025-2026 is no longer well described as a succession of new Adam variants. The design space has expanded from coordinates to matrices and layers, from fixed training horizons to policies over time, and from mathematical update rules to state representations that must survive sharding and low-precision computation. This survey organizes recent optimizers and training optimization methods along four largely independent axes: temporal estimation, update geometry, horizon management, and representation and systems. It connects the spectral normalization of Muon, the

Read original source ↗ Open in workspace

recordType

paper

region

Global