TabKD: Tabular Knowledge Distillation through Interaction Diversity of Learned Feature Bins

The paper introduces TabKD, a novel approach for data-free knowledge distillation specifically designed for tabular data, addressing the limitations of existing methods by focusing on feature interaction diversity. TabKD utilizes adaptive feature bins aligned with teacher decision boundaries to generate synthetic queries that enhance pairwise interaction coverage, achieving superior student-teacher agreement across 14 out of 16 configurations on four benchmark datasets and outperforming five state-of-the-art baselines. This framework emphasizes the importance of interaction coverage in improving distillation quality, providing a new avenue for practitioners aiming to compress tabular models while maintaining predictive performance.