Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework

The paper presents a framework for multilabel emotion annotation that addresses the subjectivity of emotion labels by employing soft vote-share labels instead of traditional hard labels. It evaluates the impact of various aggregation methods on agreement estimates and emotion classifiers, demonstrating that soft supervision can capture annotator variance and improve model predictions, especially in scenarios with inherent ambiguity. This work is significant for practitioners in NLP as it offers insights into designing and evaluating emotion datasets, ultimately enhancing the robustness of emotion classification systems.