On the Expressive Power of Weight Quantization in Large Language Models

The paper presents a theoretical analysis of weight quantization in large language models, establishing that 1.58 bits is the minimum precision for effective weight quantization. It demonstrates that as quantization bits decrease, the expressive capacity of models diminishes polynomially, highlighting the trade-off between model compression and performance degradation. These insights are crucial for practitioners focused on optimizing model efficiency while maintaining expressive power in LLMs.