Behavior is how brains act on the world, and measuring it precisely is central to neuroscience, medicine and many other fields. Deep learning has made tracking bodies in video routine, yet most behavioral models still map short clips to a fixed set of labels at a single timescale. This misses two defining properties of behavior. It is embodied: the body shapes how every action unfolds. And it is hierarchical: movements nest within actions, and actions within activities. In this talk, I will present our work on learning and benchmarking these properties. Hierarchical masked autoencoders recover this structure from pose alone. Multimodal large language models describe behavior flexibly in natural language, and training them on hard negatives makes them state-of-the-art fine-grained action recognizers. Yet new multimodal datasets and benchmarks show that even the best models misread how the body moves, and simply adding pose or motion as input helps little. These results suggest that the next generation of behavioral models should treat the body as a modality in its own right.
These seminars are being made possible through the support of the CFM-ENS Chair « Modèles et Sciences des Données ».
The organizers: Giulio Biroli, Alex Cayco Gajic, Bruno Loureiro, Stéphane Mallat, Gabriel Peyré.