{"id":15,"date":"2026-10-16T17:53:18","date_gmt":"2026-10-17T00:53:18","guid":{"rendered":"https:\/\/demos.canvaswp.com\/academic\/portfolio\/a-mathematical-model-of-self-attention-dynamics\/"},"modified":"2026-10-16T17:53:18","modified_gmt":"2026-10-17T00:53:18","slug":"a-mathematical-model-of-self-attention-dynamics","status":"publish","type":"canvas_portfolio","link":"https:\/\/demos.canvaswp.com\/academic\/portfolio\/a-mathematical-model-of-self-attention-dynamics\/","title":{"rendered":"A Mathematical Model of Self-Attention Dynamics"},"content":{"rendered":"<p class=\"wp-block-paragraph\">This paper models a transformer&#8217;s self-attention layers as an interacting particle system evolving on the sphere, with depth playing the role of time.<\/p><h2 class=\"wp-block-heading\">Main results<\/h2><p class=\"wp-block-paragraph\">We prove that tokens cluster as depth grows, characterise the limiting configurations with tools from optimal transport, and give quantitative convergence rates that agree with what is observed in trained models.<\/p>","protected":false},"excerpt":{"rendered":"<p>We study the dynamics of self-attention as an interacting particle system and show that tokens evolve toward a small number of clusters as depth increases. Using tools from optimal transport, we characterise the limiting configuration and prove quantitative convergence rates that match the behaviour observed empirically in large trained transformer architectures.<\/p>\n","protected":false},"featured_media":0,"comment_status":"open","ping_status":"closed","template":"","meta":{"_canvas_portfolio_client":"","_canvas_portfolio_byline":"Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)","_canvas_portfolio_skills":"","_canvas_portfolio_url":"","_canvas_portfolio_date":"2023","_canvas_portfolio_cover":0,"_canvas_portfolio_gallery":"","_canvas_portfolio_card_images":"","_canvas_portfolio_video_file":0,"_canvas_portfolio_video_poster":0,"_canvas_portfolio_media_height":"","_canvas_portfolio_card_wide":"","_canvas_portfolio_card_aspect":"","_canvas_portfolio_card_slider_speed":"","_canvas_portfolio_card_slider_pause":"","_canvas_portfolio_card_slider_animation":"","_canvas_portfolio_layout":"","_canvas_portfolio_columns":0,"_canvas_portfolio_primary_category":"","_canvas_acad_doi":"10.5555\/acad.2023.0412","_canvas_acad_status":"Preprint","_canvas_acad_status_color":"warning","_canvas_portfolio_before_image":0,"_canvas_portfolio_filter_classes":"","_canvas_portfolio_location":"","_canvas_portfolio_size":"","_canvas_ins_cover_art":0,"_canvas_ins_cover_benefits":"","_canvas_ins_cover_claim":"","_canvas_ins_cover_icon":0,"_canvas_ins_cover_key":"","_canvas_ins_cover_premium":"","_canvas_ins_cover_sum":"","_canvas_ins_cover_summary":"","_canvas_portfolio_city":"","_canvas_portfolio_headline":"","_canvas_portfolio_more":"","_canvas_portfolio_more_title":"","_canvas_portfolio_scope":"","_canvas_portfolio_status":"","_canvas_portfolio_type":"","_canvas_portfolio_outcome":"","_canvas_portfolio_aperture":"","_canvas_portfolio_camera":"","_canvas_portfolio_credit":"","_canvas_portfolio_iso":"","_canvas_portfolio_lens":"","_canvas_portfolio_shutter":"","_canvas_resume2_proj_badge":"","_canvas_resume2_proj_badge_icon":"","_canvas_resume2_proj_link_icon":"","_canvas_resume2_proj_link_label":"","_canvas_resume2_proj_link_url":"","_canvas_resume2_proj_tags":"","_canvas_portfolio_role":""},"canvas_portfolio_category":[13],"class_list":["post-15","canvas_portfolio","type-canvas_portfolio","status-publish","hentry","canvas_portfolio_category-papers"],"_links":{"self":[{"href":"https:\/\/demos.canvaswp.com\/academic\/wp-json\/wp\/v2\/canvas_portfolio\/15","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/demos.canvaswp.com\/academic\/wp-json\/wp\/v2\/canvas_portfolio"}],"about":[{"href":"https:\/\/demos.canvaswp.com\/academic\/wp-json\/wp\/v2\/types\/canvas_portfolio"}],"replies":[{"embeddable":true,"href":"https:\/\/demos.canvaswp.com\/academic\/wp-json\/wp\/v2\/comments?post=15"}],"wp:attachment":[{"href":"https:\/\/demos.canvaswp.com\/academic\/wp-json\/wp\/v2\/media?parent=15"}],"wp:term":[{"taxonomy":"canvas_portfolio_category","embeddable":true,"href":"https:\/\/demos.canvaswp.com\/academic\/wp-json\/wp\/v2\/canvas_portfolio_category?post=15"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}