La recherche

Across mathematics, formality results are usually established through general criteria rather than by constructing explicit equivalences. These criteria include domination, puncturing, highly-connected formality, intrinsic formality, formality descent, formality in families, purity, etc. In this article, we prove that all these criteria are instances of a single theorem, formality transmission. This unifying framework allows us to extend these criteria in several directions, beyond characteristic zero and to differential graded algebras over operads or properads, leading to diverse new formality results.

This PhD thesis presents contributions to the field of deep learning. From convolutional ResNets to Transformers, residual connections are ubiquitous in state-of-the-art deep learning models. The continuous depth analogues of residual networks, neural ODEs, have been widely adopted, but the connection between the discrete and continuous models still lacks a solid mathematical foundation. In this manuscript, we will show that for a formal correspondence between residual networks and neural ODEs to hold, the residual functions must be smooth with depth, and we will present an implicit regularization result of deep residual networks towards neural ODEs. We will then present two applications of this analogy to the design and study of new architectures. First, we will introduce a drop-in replacement for any residual network that can be trained with the same accuracy, but with much less memory. Second, by viewing the attention mechanism as an interacting particle system, where the particles are the tokens, we will study the impact of attention map normalization on the Transformer model. Finally, we will present some other contributions to Transformers: how Transformers perform in-context autoregressive learning and how to differentiably route tokens to experts in Sparse Mixture of Experts Transformers.