LLMs Prompted for Legal Context Object More: Overrefusal from Small On-Premises LLMs in Criminal Legal Context

The study examines the performance of small on-premises LLMs in legal contexts, focusing on the phenomenon of overrefusal during interactions. It reveals that authority-style prefixes can significantly increase refusal rates by 2-20 times compared to a baseline without prefixes, indicating that these models exhibit instability when faced with contextual framings typical in legal settings. This research highlights the need for careful design and evaluation of LLMs used in legal applications to mitigate bias and ensure reliable assistance.