Designed and built with care, filled with creative elements

Top
Image Alt

ENS-Data Science colloquium – Alexander Mathis (EPFL) : Hierarchy and the body: what do multimodal models understand about behavior?

  /  Évènements
Chargement Évènements

« Tous les Évènements

ENS-Data Science colloquium – Alexander Mathis (EPFL) : Hierarchy and the body: what do multimodal models understand about behavior?

3 décembre 2026 | 12h00 – 13h00

Behavior is how brains act on the world, and measuring it precisely is central to neuroscience, medicine and many other fields. Deep learning has made tracking bodies in video routine, yet most behavioral models still map short clips to a fixed set of labels at a single timescale. This misses two defining properties of behavior. It is embodied: the body shapes how every action unfolds. And it is hierarchical: movements nest within actions, and actions within activities. In this talk, I will present our work on learning and benchmarking these properties. Hierarchical masked autoencoders recover this structure from pose alone. Multimodal large language models describe behavior flexibly in natural language, and training them on hard negatives makes them state-of-the-art fine-grained action recognizers. Yet new multimodal datasets and benchmarks show that even the best models misread how the body moves, and simply adding pose or motion as input helps little. These results suggest that the next generation of behavioral models should treat the body as a modality in its own right.

These seminars are being made possible through the support of the CFM-ENS Chair « Modèles et Sciences des Données ».

The organizers: Giulio Biroli, Alex Cayco Gajic, Bruno Loureiro, Stéphane Mallat, Gabriel Peyré.

Amphi Jean Jaurès

29 rue d'Ulm
PARIS, 75005 France