– Papers
Published on: 2023
Authors: Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Status: Preprint
DOI: 10.5555/acad.2023.0412

A Mathematical Model of Self-Attention Dynamics

This paper models a transformer’s self-attention layers as an interacting particle system evolving on the sphere, with depth playing the role of time.

Main results

We prove that tokens cluster as depth grows, characterise the limiting configurations with tools from optimal transport, and give quantitative convergence rates that agree with what is observed in trained models.

More Works

On the Emergence of Clusters in Transformer Models
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Closed
DOI: 10.5555/acad.2023.0388

Abstract

We examine how token representations group together as information propagates through successive attention layers. Casting the process as a measure-valued flow, we identify the conditions under which distinct clusters form and quantify how their number depends on temperature, depth, and the spectral properties of the underlying weight matrices used.
Mean-Field Limits for Deep Residual Networks
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Info
DOI: 10.5555/acad.2023.0351

Abstract

We derive a mean-field description of deep residual networks in which the discrete layers are replaced by a continuous-time flow on the space of probability measures. This viewpoint clarifies how skip connections stabilise training and yields rigorous bounds on the gap between the finite-depth network and its idealised limiting dynamics here.
Convergence Rates in Optimal Transport Estimation
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)
Preprint
DOI: 10.5555/acad.2023.0297

Abstract

We establish finite-sample convergence rates for estimators of optimal transport maps between high-dimensional distributions. Combining minimax analysis with smoothness assumptions on the underlying densities, we obtain bounds that improve upon prior work and clarify the precise role played by dimension and regularity in the estimation problem.