On the Smallness of the Large Language Models Scaling Exponents

The article discusses the implications of scaling exponents in Large Language Models (LLMs), highlighting their indication of an unsustainable energy consumption regime. It critiques the notion that the smallness of these exponents is merely a numerical bias related to the "pedestal effect" and emphasizes that this does not resolve the sustainability concerns. Additionally, it explores how data characteristics, such as smoothness and roughness, influence scaling exponents, drawing parallels with fluid turbulence models, which may inform future model design and efficiency considerations for practitioners.