Legal Reasoning Is Not Lawyering: Rethinking Legal Benchmarks for Pro Se Access to Justice

The article critiques current legal AI benchmarks, arguing they primarily assess large language models (LLMs) under idealized conditions set by legal experts, rather than the more challenging scenarios faced by pro se litigants. It highlights the need for benchmarks that evaluate model robustness against noisy, incomplete, and error-prone inputs typical of self-represented individuals, citing issues like long-context sensitivity and hallucination. The authors advocate for developing metrics that accurately reflect model performance in these real-world conditions to ensure that claims about improving access to justice through legal AI are substantiated.