Gated MLPs as Symmetry-Broken Rank-1 Bilinear Attention

This article presents a theoretical framework that interprets conventional gated MLPs as a rank-1 approximation of a bilinear attention mechanism, highlighting the role of symmetry-breaking in their architecture. The authors demonstrate that applying nonlinearity to one factor of the bilinear attention disrupts exchange symmetry and inverse-scaling symmetry, which may elucidate the practical effectiveness of gated MLPs. This insight could influence future architectural designs in AI by optimizing attention mechanisms for better performance.