Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?

The article discusses an open problem regarding the effectiveness of the AdamW optimizer under heavy-tailed noise, which is common in large language model (LLM) pretraining. While AdamW is widely used, its theoretical foundation remains largely untested in this context, contrasting with recent findings that sign-based optimizers like Lion and Muon perform well under similar conditions. The authors propose a rigorous inquiry into AdamW's convergence properties under heavy-tailed assumptions and present preliminary results, including a positive benchmark and a mechanism highlighting how denominator memory can obscure large gradients, which is crucial for practitioners seeking to optimize LLM training in noisy environments.