StatABench: Dataset and Framework for Evaluating Statistical Analysis Capabilities of LLMs

StatABench has been introduced as a benchmark for evaluating the statistical analysis capabilities of large language models (LLMs), comprising two components: Stat-Closed with 404 questions across 18 topics and Stat-Open featuring 30 complex modeling tasks. Evaluation using the LangChain MCP framework revealed that even the advanced GPT-5.1 model only achieved 68.6% accuracy on Stat-Closed, while the best open-source model reached 60.6%, indicating significant gaps in LLM performance in statistical reasoning and modeling. This benchmark is crucial for practitioners aiming to enhance LLMs' proficiency in statistical analysis and decision-making tasks.