– Papers
Published on: 2023
Authors: Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Status: Preprint
DOI: 10.5555/acad.2023.0297

Convergence Rates in Optimal Transport Estimation

We study estimators of optimal transport maps between high-dimensional distributions from finite samples.

Main results

Combining minimax analysis with smoothness assumptions, we obtain convergence rates that improve on prior work and make the roles of dimension and regularity precise.

More Works

A Mathematical Model of Self-Attention Dynamics
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Preprint
DOI: 10.5555/acad.2023.0412

Abstract

We study the dynamics of self-attention as an interacting particle system and show that tokens evolve toward a small number of clusters as depth increases. Using tools from optimal transport, we characterise the limiting configuration and prove quantitative convergence rates that match the behaviour observed empirically in large trained transformer architectures.
On the Emergence of Clusters in Transformer Models
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Closed
DOI: 10.5555/acad.2023.0388

Abstract

We examine how token representations group together as information propagates through successive attention layers. Casting the process as a measure-valued flow, we identify the conditions under which distinct clusters form and quantify how their number depends on temperature, depth, and the spectral properties of the underlying weight matrices used.
Mean-Field Limits for Deep Residual Networks
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Info
DOI: 10.5555/acad.2023.0351

Abstract

We derive a mean-field description of deep residual networks in which the discrete layers are replaced by a continuous-time flow on the space of probability measures. This viewpoint clarifies how skip connections stabilise training and yields rigorous bounds on the gap between the finite-depth network and its idealised limiting dynamics here.