The Discrete-Log Clock: How a Transformer Learns Modular Multiplication

This study introduces the "Discrete-Log Clock," a novel approach for understanding how transformers learn modular multiplication, specifically in the context of the operation \(a \cdot b \mod 113\). By applying the multiplicative character transform instead of the standard additive discrete Fourier transform (DFT), the researchers reveal that the embedding spectrum is highly sparse, with only four key frequencies carrying significant energy and 96.9% of MLP neurons tuned to a single multiplicative frequency. This insight is crucial for practitioners as it highlights the importance of aligning analysis methods with the underlying algebraic structures of tasks, potentially improving the interpretability and efficiency of transformer models in similar applications.