LEDGER: A Long-Context Benchmark of Corporate Annual Reports for Grounded Financial Retrieval and Extraction

Researchers have released LEDGER, a new benchmark designed to test the ability of large language models to extract financial information from nearly 5,000 digitized corporate annual reports, moving beyond the plain-text regulatory filings used in most existing evaluations [1][2]. The corpus consists of 4,999 full annual reports—including figures, tables, and narrative—each labeled with 31 consolidated financial key performance indicators [1][2]. Unlike prior resources that rely on text-only SEC 10-K filings, LEDGER incorporates the complete visual and structural complexity of published shareholder documents [2]. The United States Securities and Exchange Commission was created in the aftermath of the 1929 Wall Street crash to enforce laws against market manipulation, and its required filings have long served as a testbed for language models [5]. From this data, the team derived three evaluation benchmarks of increasing difficulty. The first is a page-level KPI retrieval task with 118,048 natural-language questions and TREC-style relevance judgments [1][2]. The second is a conversational “needle-in-a-haystack” single-value lookup, and the third requires full KPI extraction from long, numerically dense reports [2]. The release includes human optical-character-recognition-quality annotations with inter-annotator agreement scores and a complete extraction, validation, and scoring toolchain [1][2]. Large language models are machine learning systems with many parameters, trained on vast amounts of text through self-supervised learning [6]. Their expanding context windows have made rigorous evaluation in finance an increasingly pressing need, according to the paper’s authors [2]. The LEDGER benchmark is designed to probe those very-long-context capabilities by forcing models to locate and reason over specific figures buried inside hundreds of pages of mixed-format content [2]. A case study included with the release links the rhetoric found in CEO letters to post-publication market impact, demonstrating one applied research pathway the dataset enables [1][2]. The full corpus and toolchain are available for research purposes [1].