The Weight Norm Sets the Grokking Timescale: A Causal Delay Law

The study presents findings on the phenomenon of grokking in neural networks, identifying a causal relationship between weight norm and the timescale of generalization. It demonstrates that when the weight norm reaches a critical value \( W_c \), the grokking delay follows an exponential relationship with a fitted exponent near 7.5, significantly impacting training dynamics. This research is crucial for practitioners as it provides insights into optimizing training strategies and understanding the underlying mechanisms influencing generalization in LLMs.