-
Effective dynamics and critical scaling for Stochastic Gradient Descent in high dimensions – Gerard Ben Arous (New York University)
Effective dynamics and critical scaling for Stochastic Gradient Descent in high dimensions – Gerard Ben Arous (New York University)
Gerard Ben Arous (New York University) Title: Effective dynamics and critical scaling for Stochastic Gradient Descent in high dimensions Abstract: SGD in high dimension is a workhorse for high dimensional statistics and machine learning, but understanding its behavior in high dimensions is not yet a simple task. We study here the limiting 'effective' dynamics of some summary statistics for SGD in high dimensions, and find interesting and new regimes, i.e. not the expected one given by the population gradient flow. We find that a new corrector term is needed and that the phase […]