What Shapes Emergent Misalignment? Insights from Training Dynamics, Model Priors, and Data

The paper investigates emergent misalignment (EM) in machine learning models, focusing on how training dynamics, model priors, and data influence alignment during fine-tuning. It reveals that while in-domain training loss does not strongly correlate with improved out-of-domain alignment scores, there are predictive signals in activation patterns from pre-trained models that can inform fine-tuning outcomes. This research is significant for practitioners as it highlights the complexities of fine-tuning strategies and the importance of understanding activation shifts to mitigate misalignment in AI models.