Fin-RATE: A Real-world Financial Analytics and Tracking Evaluation Benchmark for LLMs on SEC Filings

Fin-RATE is a new benchmark designed for evaluating Large Language Models (LLMs) on U.S. SEC filings, addressing the need for a more comprehensive assessment of LLM performance in financial analysis. It simulates the workflows of financial analysts through three evaluation pathways: detail-oriented reasoning, cross-entity comparisons, and longitudinal tracking. Benchmarking 17 LLMs revealed significant performance drops—up to 18.60%—as tasks increased in complexity, highlighting issues such as hallucinations and mismatches that existing benchmarks do not adequately address, making it crucial for practitioners to understand the limitations of LLMs in real-world financial contexts.