– Papers
Published on: 2023
Authors: Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Status: Closed
DOI: 10.5555/acad.2023.0388

On the Emergence of Clusters in Transformer Models

We follow token representations through successive attention layers and describe their evolution as a measure-valued flow.

Main results

The paper identifies when distinct clusters form and how their number depends on temperature, depth and the spectrum of the weight matrices.

More Works

A Mathematical Model of Self-Attention Dynamics
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Preprint
DOI: 10.5555/acad.2023.0412

Abstract

We study the dynamics of self-attention as an interacting particle system and show that tokens evolve toward a small number of clusters as depth increases. Using tools from optimal transport, we characterise the limiting configuration and prove quantitative convergence rates that match the behaviour observed empirically in large trained transformer architectures.
Mean-Field Limits for Deep Residual Networks
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Info
DOI: 10.5555/acad.2023.0351

Abstract

We derive a mean-field description of deep residual networks in which the discrete layers are replaced by a continuous-time flow on the space of probability measures. This viewpoint clarifies how skip connections stabilise training and yields rigorous bounds on the gap between the finite-depth network and its idealised limiting dynamics here.
Convergence Rates in Optimal Transport Estimation
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Preprint
DOI: 10.5555/acad.2023.0297

Abstract

We establish finite-sample convergence rates for estimators of optimal transport maps between high-dimensional distributions. Combining minimax analysis with smoothness assumptions on the underlying densities, we obtain bounds that improve upon prior work and clarify the precise role played by dimension and regularity in the estimation problem.