Beyond Questions: Evaluating What Large Language Models (Actually) Know

A team of researchers has proposed a new method for evaluating the factual knowledge of large language models, arguing that current benchmarks are limited by their reliance on predefined questions. The new approach, detailed in a paper submitted 26 May 2026, uses open-ended prompts to assess what information models naturally express. [1] The paradigm, called open knowledge evaluation, is designed to address what the authors describe as an "availability bias" in existing tests. Standard knowledge benchmarks typically ask narrow questions, such as "What is the birth date of M.L. King?," which only measures knowledge that test designers explicitly seek. [1] The new method instead uses prompts like "Tell me everything you know about M.L. King" to capture the knowledge a model chooses to surface on its own. [1] To implement this approach, the researchers created BeQu, short for "Beyond Questions." The benchmark consists of 10,000 entities, each paired with reference corpora used to verify the accuracy of statements generated by a model. [1] The paper analyzes how factors including reasoning effort, model scale, prompt format, and knowledge domain affect the knowledge that language models express. [1] Large language models, or LLMs, are neural networks trained on vast text corpora and serve as the foundation for modern chatbots and AI assistants. [3] Their capabilities are typically measured through standardized benchmarks, which provide datasets and metrics for tasks like question answering and text generation. [5] However, biased or inaccurate training data can make an LLM's output less reliable, making robust evaluation critical. [3] The release of the BeQu benchmark comes amid rapid evolution in the LLM landscape. Meta AI, for instance, has released multiple generations of its Llama model family, with sizes ranging from 1 billion to 2 trillion parameters, and introduced its successor, Muse Spark, in April 2026. [4] The BeQu paper's authors have made their data and a public leaderboard available on the project's website and GitHub repository. [1]