BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Département de mathématiques et applications - ECPv6.2.2//NONSGML v1.0//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
X-WR-CALNAME:Département de mathématiques et applications
X-ORIGINAL-URL:https://www.math.ens.psl.eu
X-WR-CALDESC:évènements pour Département de mathématiques et applications
REFRESH-INTERVAL;VALUE=DURATION:PT1H
X-Robots-Tag:noindex
X-PUBLISHED-TTL:PT1H
BEGIN:VTIMEZONE
TZID:Europe/Paris
BEGIN:DAYLIGHT
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
TZNAME:CEST
DTSTART:20240331T010000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
TZNAME:CET
DTSTART:20241027T010000
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTART;TZID=Europe/Paris:20240404T120000
DTEND;TZID=Europe/Paris:20240404T140000
DTSTAMP:20260515T101120
CREATED:20240502T082427Z
LAST-MODIFIED:20240502T082427Z
UID:17436-1712232000-1712239200@www.math.ens.psl.eu
SUMMARY:ENS-Data Science colloquium - Lénaïc Chizat (EPFL)
DESCRIPTION:04 Avril 2024\, Lénaïc Chizat (EPFL)\nTitle: A Formula for Feature Learning in Large Neural Networks\nAbstract: Deep learning succeeds by doing hierarchical feature learning\, but tuning hyperparameters such as initialization scales\, learning rates\, etc.\, only give indirect control over this behavior. This calls for theoretical tools to predict\, measure and control feature learning. In this talk\, we will first review various theoretical advances (signal propagation\, infinite width dynamics\, etc) that have led to a better understanding of the subtle impact of hyperparameters and architectural choices on the training dynamics. We will then introduce a formula which\, in any architecture\, quantifies feature learning in terms of more tractable quantities: statistics of the forward and backward passes\, and a notion of alignment between the feature updates and the backward pass which captures an important aspect of the nature of feature learning. This formula suggests normalization rules for the forward and backward passes and for the layer-wise learning rates. To illustrate these ideas\, I will discuss the feature learning behavior of ReLU MLPs and ResNets in the infinite width and depth limit.
URL:https://www.math.ens.psl.eu/evenement/ens-data-science-colloquium-lenaic-chizat-epfl/
LOCATION:Amphi Jaurès (29 Rue d’Ulm)
CATEGORIES:Séminaire Data de l’ENS
END:VEVENT
END:VCALENDAR