Peeking Inside LLMs: Leveraging Internal Artifacts of LLMs for Enhancing Reliability in Legal Classification

This paper presents a method for improving the reliability of Large Language Models (LLMs) in legal classification tasks by utilizing internal artifacts to detect incorrect outputs. The authors developed classifiers that leverage these internal features and evaluated their effectiveness on bail decision prediction and statute violation prediction tasks, demonstrating that these internal indicators can significantly enhance the accuracy of LLM-based systems. This work is crucial for practitioners in the legal domain, as it addresses the challenge of hallucination in LLMs, thereby increasing trust in AI applications for high-stakes decisions.