DECSELFMASK: Leveraging Unlabeled Text via Self-Relevance-Guided Masking for Decoder-Only Classification

The article presents DecSelfMask, a novel self-learning approach designed for decoder-only models to enhance classification performance using unlabeled text. It introduces a relevance-guided masking strategy that utilizes relevance attribution methods to create self-supervised training examples from unannotated data, demonstrating consistent improvements across 136 tasks on a dataset of 1.9M clinical notes, with notable performance gains over standard fine-tuning and synthetic label generation methods. This approach is significant for practitioners as it provides a scalable method to leverage unlabeled data, particularly in domains with scarce annotated datasets, thereby improving model performance in real-world applications.