SpikeDecoder: Realizing the GPT Architecture with Spiking Neural Networks

The paper introduces SpikeDecoder, a fully spiking neural network (SNN) implementation of the Transformer decoder block aimed at natural language processing. It explores the challenges of training SNNs and analyzes the effects of replacing various artificial neural network (ANN) components with spiking alternatives, finding that the SNN-based decoder can reduce theoretical energy consumption by 87% to 93% compared to traditional ANN models. This development is significant for practitioners as it offers a more energy-efficient approach to NLP tasks while addressing the training difficulties associated with SNNs.