SOURCE-LINKED INTELLIGENCE

Structured Features Overfit Where Random Features Grok

arXiv · AI, language, vision and robotics · article · Sep 14, 2026 · UTC

Xu, Vardi and Safran (ICML 2026) prove that over-parameterized ridge regression over an unstructured random Gaussian feature map groks, with the delay between memorization and generalization growing as $1/λ$ in the weight decay. We show that on a structured feature map the same delay does not appear. For a band-limited Fourier feature map over $\mathbb{Z}_p^2$ carrying a single-character target that lies inside the expressible class, enlarging the band at fixed positive weight decay drives peak held-out accuracy monotonically from $1.00$ to $0.07$, with no memorize-then-generalize regime anywh

Read original source ↗ Open in workspace

recordType

paper

region

Global