All Routes Lead to Collapse

The paper presents a study on pathologies in transformer models, specifically addressing attention sinks, representation collapse, and norm stratification, asserting they arise from content-based routing with fixed similarity metrics rather than being exclusive to attention mechanisms. It introduces a reframing identity for softmax attention as Boltzmann-weighted aggregation that overlooks key magnitude, leading to concentration and collapse of routed representations across various routing mechanisms tested, including softmax attention, graph attention, and state-space models. This research highlights the limitations of conventional routing metrics and suggests that practitioners should consider the implications of routing mechanisms on representation quality when designing and implementing AI models.