SOURCE-LINKED INTELLIGENCE

Convergence rates for the RMSprop optimizer with full control of the hyperparameters

arXiv · AI, language, vision and robotics · article · Aug 31, 2026 · UTC

Popular adaptive stochastic gradient descent (SGD) methods to train artificial intelligence (AI) systems include the RMSprop, the Adam, and the AdamW optimizers, where the adaptivity parts in Adam and AdamW basically just coincide with RMSprop. Such adaptive methods involve several hyperparameters including the regularization parameter $ε$ (which ensures that one does not divide by 0 and is often chosen to be very close to zero such as $10^{-8}$ in PyTorch by default) and the second moment decay parameter $β$ (which is often chosen to be very close to $1$ such as 0.99 (RMSprop) and 0.999 (Adam

Read original source ↗ Open in workspace

recordType

paper

region

Global