TALAS: Teacher-Anchored Layer Alignment with Adaptive Sharpness-Aware Minimization for Embedding Distillation

The paper introduces TALAS (Teacher-Anchored Layer Alignment with Adaptive Sharpness-Aware Minimization), a novel framework for knowledge distillation that optimizes the training of student models by selectively aligning only the upper layers with teacher embeddings and employing Layer-Aligned Self-Distillation for lower layers. By integrating Adaptive Sharpness-Aware Minimization, TALAS enhances model generalization while reducing computational costs and memory usage. Empirical results show that TALAS outperforms existing distillation methods on standard sentence embedding benchmarks, making it a significant advancement for practitioners in model compression and efficiency.