– Papers
Published on: 2023
Authors: Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Status: Info
DOI: 10.5555/acad.2023.0351

Mean-Field Limits for Deep Residual Networks

We replace the discrete layers of a deep residual network with a continuous-time flow on the space of probability measures.

Main results

The mean-field view explains how skip connections stabilise training and gives rigorous bounds between finite-depth networks and their limiting dynamics.

More Works

A Mathematical Model of Self-Attention Dynamics
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Preprint
DOI: 10.5555/acad.2023.0412

Abstract

We study the dynamics of self-attention as an interacting particle system and show that tokens evolve toward a small number of clusters as depth increases. Using tools from optimal transport, we characterise the limiting configuration and prove quantitative convergence rates that match the behaviour observed empirically in large trained transformer architectures.
On the Emergence of Clusters in Transformer Models
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Closed
DOI: 10.5555/acad.2023.0388

Abstract

We examine how token representations group together as information propagates through successive attention layers. Casting the process as a measure-valued flow, we identify the conditions under which distinct clusters form and quantify how their number depends on temperature, depth, and the spectral properties of the underlying weight matrices used.
Convergence Rates in Optimal Transport Estimation
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Preprint
DOI: 10.5555/acad.2023.0297

Abstract

We establish finite-sample convergence rates for estimators of optimal transport maps between high-dimensional distributions. Combining minimax analysis with smoothness assumptions on the underlying densities, we obtain bounds that improve upon prior work and clarify the precise role played by dimension and regularity in the estimation problem.