LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake

A new benchmark called LakeQA has been introduced to test how well large language models can search for and reason over information scattered across massive, heterogeneous data lakes, according to a paper submitted to arXiv on June 9, 2026 [1]. The benchmark is built on a heterogeneous collection of approximately 9.5 TB of text resources drawn from Wikipedia and open-source government data, spanning both structured and unstructured formats [1][2]. Each task in LakeQA requires long-horizon multi-hop reasoning with implicit intermediate steps, meaning an AI agent must independently discover the correct documents and then compose evidence across multiple sources to produce an answer [1][2]. To ensure task quality, each sample was annotated by at least one Ph.D.-level expert [1][2]. LakeQA is designed to address a gap in current evaluation methods. While large language models, which are machine learning models trained on vast amounts of text for natural language processing tasks [7], have shown rapid progress in reading-based question answering where evidence is explicitly provided, real-world questions are often not paired with accurate evidence documents [1][2]. The useful evidence resides in massive data lakes, making search a prerequisite for answering [1][2]. Experimental results on seven frontier LLMs demonstrate that LakeQA is challenging [1][2]. GPT-5.2 achieved only an exact-match score of 18.37% on the benchmark [1][2]. The paper positions LakeQA as a realistic testbed for developing LLM agents that can both find and analyze data in modern data lakes [1][2]. The paper appears on arXiv, a preprint server that has integrated with Hugging Face Spaces to allow authors and the community to link interactive demos directly from a paper's abstract page [3][4]. This integration enables users to try out machine learning models in a browser without writing any code, using tools such as Gradio and Streamlit [3][4][5]. The broader field of retrieval-augmented generation, which combines language models with information retrieval, was advanced by researchers including Douwe Kiela, who co-authored the foundational paper on RAG in 2020 while at Meta AI and later served as Head of Research at Hugging Face [8].