– Papers
Published on: 2023
Authors: Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Status: Preprint
DOI: 10.5555/acad.2023.0297
Convergence Rates in Optimal Transport Estimation
We study estimators of optimal transport maps between high-dimensional distributions from finite samples.
Main results
Combining minimax analysis with smoothness assumptions, we obtain convergence rates that improve on prior work and make the roles of dimension and regularity precise.
More Works
A Mathematical Model of Self-Attention Dynamics
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Preprint
DOI: 10.5555/acad.2023.0412Abstract
We study the dynamics of self-attention as an interacting particle system and show that tokens evolve toward a small number of clusters as depth increases. Using tools from optimal transport, we characterise the limiting configuration and prove quantitative convergence rates that match the behaviour observed empirically in large trained transformer architectures.On the Emergence of Clusters in Transformer Models
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Closed
DOI: 10.5555/acad.2023.0388Abstract
We examine how token representations group together as information propagates through successive attention layers. Casting the process as a measure-valued flow, we identify the conditions under which distinct clusters form and quantify how their number depends on temperature, depth, and the spectral properties of the underlying weight matrices used.Mean-Field Limits for Deep Residual Networks
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Info
DOI: 10.5555/acad.2023.0351