GRINQH: Graded Input-based Quantization Hierarchy for Efficient LLM Generation

GRINQH (GRaded INput-based Quantization Hierarchy) is a weight-only post-training quantization framework designed to enhance LLM decoding efficiency by addressing the asymmetry between compute-bound prefill and memory-bound decoding stages. It dynamically assigns weight channels to varying precision levels based on activation magnitudes, allowing for flexible average bit widths, and has demonstrated superior performance on Llama3 and Qwen3 models, achieving effective 2-bit generation while surpassing existing fixed and mixed-precision methods. This development is significant for practitioners as it establishes a new Pareto frontier for balancing generation quality and inference speed, particularly in resource-constrained environments.