Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails

The paper presents a systematic study on Knowledge Distillation (KD) in the post-training phase, specifically utilizing the large-scale Tulu 3 dataset. It finds that KD outperforms supervised fine-tuning in low-data scenarios, with diminishing returns as data increases; however, distillation from a strong instruction-tuned teacher can enhance performance even with abundant data. The authors propose a two-stage KD strategy that combines synthetic teacher-labeled data and human refinement, offering a practical approach for developing efficient models in resource-constrained settings.