ITNet: A Learnable Integral Transform That Subsumes Convolution, Attention, and Recurrence

The Integral Transform Network (ITNet) has been introduced as a unified architecture that subsumes convolution, attention, and recurrence through a learnable integral transform. ITNet employs a small MLP for its kernel, allowing it to adaptively model pairwise interactions, thus achieving performance that matches or exceeds specialized models on benchmarks like ImageNet-1K and GLUE. This advancement is significant for practitioners as it provides a single framework that can efficiently replicate the capabilities of multiple architectures, streamlining the development process in AI applications.