SOURCE-LINKED INTELLIGENCE

When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay

arXiv · AI, language, vision and robotics · article · Sep 8, 2026 · UTC

Normalization renders large parts of neural networks effectively scale invariant, inducing a hidden feedback loop in which learning-rate schedules and weight decay interact through the parameter norm to control the effective step taken by the optimizer. We show that this interaction is governed by an exact discrete-time law: a single scalar quantity captures all schedule and decay forcing, while norm growth induces an opposing geometric self-quenching effect. This yields a sharp boundary that cleanly separates contraction- and expansion-dominated effective learning rate regimes. To understand

Read original source ↗ Open in workspace

recordType

paper

region

Global