Demographic Metadata as Construct-Irrelevant Noise in DistilBERT-Based Automated Essay Scoring

This study investigates the impact of integrating demographic metadata into a DistilBERT-based Automated Essay Scoring (AES) system using a naive multimodal fusion strategy. Evaluating models on the ASAP 2.0 dataset, the baseline model achieved a Quadratic Weighted Kappa (QWK) of 0.727, which declined to 0.656 when demographic data was included, indicating a significant degradation in predictive accuracy and increased validation loss. These findings highlight the detrimental effects of demographic metadata on model performance and scoring bias, providing critical insights for practitioners aiming to develop fair and effective AES systems.