Gefen: Optimized Stochastic Optimizer

Gefen is a new memory-efficient optimizer introduced as a drop-in replacement for AdamW, reducing memory usage by approximately 8x while maintaining performance. It achieves this by sharing second-moment estimates across parameter blocks and quantizing the first moment using a learned codebook, resulting in a reduction of 6.5 GiB per billion parameters. This optimization allows for larger microbatches and improved throughput during training, making it particularly beneficial for practitioners working with large models or batch sizes in frameworks like FSDP and DDP.