Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courts

The article introduces TF-RefusalBench, a multilingual benchmark designed to assess over-alignment in LLMs used for criminal law translation and summarization, consisting of 5,200 prompts in French, German, Italian, and English. It highlights that over-alignment, characterized by excessive refusals due to model guardrails, significantly affects the performance of LLMs in sensitive legal contexts. The study demonstrates that while prompting can mitigate refusals, abliteration techniques can effectively eliminate refusals with minimal impact on task performance, providing insights for practitioners aiming to deploy LLMs in legal settings.