One Year Later...The Harms Persist, But So Do We!

This study evaluates six proprietary large language models (LLMs) in the context of mental health, assessing their performance across 16 DSM-5 conditions using four adversarial attack variants. An eight-dimension harm taxonomy and a multi-dimensional evaluation framework were introduced, revealing that safeguards are effective primarily for suicide and self-harm, while models failed to protect against risks associated with eating disorders, substance use, and major depressive disorder, with failure rates reaching 100%. The findings underscore the urgent need for clearly defined harm categories and robust safety measures in the deployment of LLMs in sensitive applications to mitigate risks to vulnerable populations.