– Papers
Published on: 2023
Authors: Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Status: Preprint
DOI: 10.5555/acad.2023.0412
A Mathematical Model of Self-Attention Dynamics
This paper models a transformer’s self-attention layers as an interacting particle system evolving on the sphere, with depth playing the role of time.
Main results
We prove that tokens cluster as depth grows, characterise the limiting configurations with tools from optimal transport, and give quantitative convergence rates that agree with what is observed in trained models.
More Works
On the Emergence of Clusters in Transformer Models
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Closed
DOI: 10.5555/acad.2023.0388Abstract
We examine how token representations group together as information propagates through successive attention layers. Casting the process as a measure-valued flow, we identify the conditions under which distinct clusters form and quantify how their number depends on temperature, depth, and the spectral properties of the underlying weight matrices used.Mean-Field Limits for Deep Residual Networks
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Info
DOI: 10.5555/acad.2023.0351Abstract
We derive a mean-field description of deep residual networks in which the discrete layers are replaced by a continuous-time flow on the space of probability measures. This viewpoint clarifies how skip connections stabilise training and yields rigorous bounds on the gap between the finite-depth network and its idealised limiting dynamics here.Convergence Rates in Optimal Transport Estimation
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Preprint
DOI: 10.5555/acad.2023.0297