Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation

The article introduces Taylor-Calibrate, a novel initialization method for hybrid Gated DeltaNet (GDN) models aimed at improving the distillation process from pretrained Transformers. By utilizing Taylor-guided teacher attention statistics for setting key parameters and aligning layers, this method significantly enhances the performance of zero-shot students, achieving up to an 88x improvement in efficiency and requiring 4.9x–9.2x fewer training tokens compared to traditional conversion techniques. This advancement is crucial for AI practitioners as it streamlines the process of creating efficient models for long-context inference while maintaining high-quality outputs.