SOURCE-LINKED INTELLIGENCE

Beyond Quadratic Loss: The Stability Phase Diagram of Adam

arXiv · AI, language, vision and robotics · article · Sep 16, 2026 · UTC

Loss spikes are recurrent instabilities in neural-network training and can arise from multiple mechanisms. For Adam in particular, macroscopic loss spikes have been linked to optimizer dynamics, yet how its two momentum timescales govern them remains unclear. We investigate this dependence by mapping training dynamics across the $(β_1,β_2)$ plane. Across a range of model--task settings, an approximately linear boundary, $1-β_2=C(1-β_1)$, separates spiky from non-spiky dynamics, whereas a one-dimensional quadratic loss produces approximately cubic slope. A one-dimensional superquadratic loss $L

Read original source ↗ Open in workspace

recordType

paper

region

Global