– Papers
Published on: 2023
Authors: Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Status: Closed
DOI: 10.5555/acad.2023.0388
On the Emergence of Clusters in Transformer Models
We follow token representations through successive attention layers and describe their evolution as a measure-valued flow.
Main results
The paper identifies when distinct clusters form and how their number depends on temperature, depth and the spectrum of the weight matrices.
More Works
A Mathematical Model of Self-Attention Dynamics
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Preprint
DOI: 10.5555/acad.2023.0412Abstract
We study the dynamics of self-attention as an interacting particle system and show that tokens evolve toward a small number of clusters as depth increases. Using tools from optimal transport, we characterise the limiting configuration and prove quantitative convergence rates that match the behaviour observed empirically in large trained transformer architectures.Mean-Field Limits for Deep Residual Networks
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Info
DOI: 10.5555/acad.2023.0351Abstract
We derive a mean-field description of deep residual networks in which the discrete layers are replaced by a continuous-time flow on the space of probability measures. This viewpoint clarifies how skip connections stabilise training and yields rigorous bounds on the gap between the finite-depth network and its idealised limiting dynamics here.Convergence Rates in Optimal Transport Estimation
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Preprint
DOI: 10.5555/acad.2023.0297