Scaling Laws for Task-Specific LLM Distillation

The paper presents empirical scaling laws for the distillation of task-specific large language models (LLMs), focusing on the trade-offs between in-domain and general knowledge performance as influenced by dataset size, compression ratio, and supervision format. It introduces a blended chain-of-thought supervision loss to enhance distillation stability and compares logit-based and LoRA-based approaches under iterative structural pruning, revealing that supervision format significantly impacts performance retention during compression. The authors release the FinHeadlineMix dataset and provide practical guidelines, offering a framework for practitioners to make informed decisions on domain-specific LLM compression strategies.