{"id":16,"date":"2026-10-16T17:53:18","date_gmt":"2026-10-17T00:53:18","guid":{"rendered":"https:\/\/demos.canvaswp.com\/academic\/portfolio\/on-the-emergence-of-clusters-in-transformer-models\/"},"modified":"2026-10-16T17:53:18","modified_gmt":"2026-10-17T00:53:18","slug":"on-the-emergence-of-clusters-in-transformer-models","status":"publish","type":"canvas_portfolio","link":"https:\/\/demos.canvaswp.com\/academic\/portfolio\/on-the-emergence-of-clusters-in-transformer-models\/","title":{"rendered":"On the Emergence of Clusters in Transformer Models"},"content":{"rendered":"<p class=\"wp-block-paragraph\">We follow token representations through successive attention layers and describe their evolution as a measure-valued flow.<\/p><h2 class=\"wp-block-heading\">Main results<\/h2><p class=\"wp-block-paragraph\">The paper identifies when distinct clusters form and how their number depends on temperature, depth and the spectrum of the weight matrices.<\/p>","protected":false},"excerpt":{"rendered":"<p>We examine how token representations group together as information propagates through successive attention layers. Casting the process as a measure-valued flow, we identify the conditions under which distinct clusters form and quantify how their number depends on temperature, depth, and the spectral properties of the underlying weight matrices used.<\/p>\n","protected":false},"featured_media":0,"comment_status":"open","ping_status":"closed","template":"","meta":{"_canvas_portfolio_client":"","_canvas_portfolio_byline":"Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet (2023)","_canvas_portfolio_skills":"","_canvas_portfolio_url":"","_canvas_portfolio_date":"2023","_canvas_portfolio_cover":0,"_canvas_portfolio_gallery":"","_canvas_portfolio_card_images":"","_canvas_portfolio_video_file":0,"_canvas_portfolio_video_poster":0,"_canvas_portfolio_media_height":"","_canvas_portfolio_card_wide":"","_canvas_portfolio_card_aspect":"","_canvas_portfolio_card_slider_speed":"","_canvas_portfolio_card_slider_pause":"","_canvas_portfolio_card_slider_animation":"","_canvas_portfolio_layout":"","_canvas_portfolio_columns":0,"_canvas_portfolio_primary_category":"","_canvas_acad_doi":"10.5555\/acad.2023.0388","_canvas_acad_status":"Closed","_canvas_acad_status_color":"danger","_canvas_portfolio_before_image":0,"_canvas_portfolio_filter_classes":"","_canvas_portfolio_location":"","_canvas_portfolio_size":"","_canvas_ins_cover_art":0,"_canvas_ins_cover_benefits":"","_canvas_ins_cover_claim":"","_canvas_ins_cover_icon":0,"_canvas_ins_cover_key":"","_canvas_ins_cover_premium":"","_canvas_ins_cover_sum":"","_canvas_ins_cover_summary":"","_canvas_portfolio_city":"","_canvas_portfolio_headline":"","_canvas_portfolio_more":"","_canvas_portfolio_more_title":"","_canvas_portfolio_scope":"","_canvas_portfolio_status":"","_canvas_portfolio_type":"","_canvas_portfolio_outcome":"","_canvas_portfolio_aperture":"","_canvas_portfolio_camera":"","_canvas_portfolio_credit":"","_canvas_portfolio_iso":"","_canvas_portfolio_lens":"","_canvas_portfolio_shutter":"","_canvas_resume2_proj_badge":"","_canvas_resume2_proj_badge_icon":"","_canvas_resume2_proj_link_icon":"","_canvas_resume2_proj_link_label":"","_canvas_resume2_proj_link_url":"","_canvas_resume2_proj_tags":"","_canvas_portfolio_role":""},"canvas_portfolio_category":[13],"class_list":["post-16","canvas_portfolio","type-canvas_portfolio","status-publish","hentry","canvas_portfolio_category-papers"],"_links":{"self":[{"href":"https:\/\/demos.canvaswp.com\/academic\/wp-json\/wp\/v2\/canvas_portfolio\/16","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/demos.canvaswp.com\/academic\/wp-json\/wp\/v2\/canvas_portfolio"}],"about":[{"href":"https:\/\/demos.canvaswp.com\/academic\/wp-json\/wp\/v2\/types\/canvas_portfolio"}],"replies":[{"embeddable":true,"href":"https:\/\/demos.canvaswp.com\/academic\/wp-json\/wp\/v2\/comments?post=16"}],"wp:attachment":[{"href":"https:\/\/demos.canvaswp.com\/academic\/wp-json\/wp\/v2\/media?parent=16"}],"wp:term":[{"taxonomy":"canvas_portfolio_category","embeddable":true,"href":"https:\/\/demos.canvaswp.com\/academic\/wp-json\/wp\/v2\/canvas_portfolio_category?post=16"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}