Nyströmformer: Approximating self-attention in linear time and memory via the Nyström method

The paper introduces Nyströmformer, a novel architecture that approximates self-attention mechanisms using the Nyström method, achieving linear time and memory complexity. By leveraging low-rank approximations, Nyströmformer reduces the quadratic scaling of traditional self-attention in transformer models, enabling efficient processing of longer sequences. This advancement is significant for practitioners aiming to deploy transformers in resource-constrained environments or for tasks requiring real-time processing of extensive datasets.