Density Field State Space Models: 1-Bit Distillation, Efficient Inference, and Knowledge Organization in Mamba-2

The article introduces Density Field State Space Models (DF-SSM), a new framework that compresses the Mamba-2 model (1.3B parameters) to a 278 MB size using 1-bit distillation and int8 low-rank correction, achieving a 21.4x speedup in inference on GPU. The distillation process requires only 32M tokens and 6 hours on a single A100 GPU, while the resulting model maintains performance within 2-4 percentage points of the larger BitMamba-2 model. This work is significant for practitioners as it presents an optimized inference pipeline and insights into knowledge organization, highlighting a structured approach to model compression and efficiency without substantial loss in performance.