CombEval: A Framework for Evaluating Combinatorial Counting in Large Language Models

A new benchmark called CombEval offers a dynamic method to test how well large language models handle combinatorial counting, according to a paper submitted in 2026 [1]. The framework generates problems with exact, solver-verified answers, moving beyond static test collections [2]. The benchmark, detailed on arXiv, represents each counting problem as a typed Cofola specification that defines entities, combinatorial objects, dependencies, and constraints [1]. This structure allows for the controlled generation of natural-language problems. Researchers can systematically vary the object type, the scale of entities involved, the number of constraints, and the required reasoning depth [2]. The authors evaluated 11 LLMs under both direct questioning and code-augmented settings [1]. The results showed that the models consistently struggled with ordered objects, indistinguishable elements, relatively positional constraints, and nested object dependencies [2]. An error analysis further identified specific failures in constraint interpretation and the application of core counting principles [1]. The release of CombEval comes as the field of large language models continues to expand rapidly. LLMs are machine learning models with a vast number of parameters, trained on massive text corpora for tasks like language generation [7]. Companies such as DeepSeek, a Chinese AI firm founded in 2023, have developed models like DeepSeek-R1 that rival offerings from OpenAI, reportedly at a fraction of the training cost [6]. The CombEval paper’s code and generated benchmark suites have been made publicly available on GitHub [2]. The project’s arXiv page also links to a demo tab, a feature resulting from a collaboration between arXiv and Hugging Face that allows users to try interactive machine learning demos directly from a paper’s abstract page [3][4]. This integration, launched in 2022, enables researchers and the community to build and share open-source demos using tools like Gradio and Streamlit, increasing the reproducibility and accessibility of research [3][5].