La recherche

We show, under a smoothness assumption, that probability measures whose exponential tilts have a uniformly bounded variance are given by a strongly convex potential, up to a bounded perturbation, yielding a partial converse to the Bakry-Émery theorem combined with Holley-Stroock's perturbation principle. A stronger version enforcing uniform convexity of the potential is obtained by considering the low-temperature regime.

This PhD thesis presents contributions to the field of deep learning. From convolutional ResNets to Transformers, residual connections are ubiquitous in state-of-the-art deep learning models. The continuous depth analogues of residual networks, neural ODEs, have been widely adopted, but the connection between the discrete and continuous models still lacks a solid mathematical foundation. In this manuscript, we will show that for a formal correspondence between residual networks and neural ODEs to hold, the residual functions must be smooth with depth, and we will present an implicit regularization result of deep residual networks towards neural ODEs. We will then present two applications of this analogy to the design and study of new architectures. First, we will introduce a drop-in replacement for any residual network that can be trained with the same accuracy, but with much less memory. Second, by viewing the attention mechanism as an interacting particle system, where the particles are the tokens, we will study the impact of attention map normalization on the Transformer model. Finally, we will present some other contributions to Transformers: how Transformers perform in-context autoregressive learning and how to differentiably route tokens to experts in Sparse Mixture of Experts Transformers.