ChLogic: Evaluating Robustness of Logical Reasoning in Chinese Expressions

A new benchmark called ChLogic reveals that large language models still struggle to maintain logical reasoning when the same problem is expressed in Chinese instead of English, according to research posted to arXiv on 16 June 2026 [1]. The benchmark, formally introduced as an English–Chinese aligned evaluation suite, tests whether models preserve logical reasoning performance when the same latent logical structure is expressed in English and diverse Chinese surface realizations [2]. It is built from formal logical templates and contains three data sets: a General aligned set derived from 60 General Propositions across nine template families; a Difficult aligned set derived from 40 Difficult Problems; and a Chinese-only set covering 15 language-specific phenomenon types [2]. Each aligned item pairs one English reference expression with five Chinese realizations [2]. Experiments were conducted on Qwen3, Ministral, and GLM models, and the results showed a persistent English–Chinese performance gap [2]. The researchers also tested a back-translation strategy, converting standard Chinese into English before feeding it to the models. On the General aligned set, back-translation often improved performance, but on the Difficult aligned set the effects were mixed: Qwen3-32B and GLM-5.1 performed worse after translation [2]. The paper concludes that Chinese surface realization, translation artifacts, and model-specific behavior jointly affect multilingual logical reasoning [2]. The work arrives as large language models are increasingly deployed across languages. LLMs are machine learning models designed for natural language processing tasks such as language generation, trained with self-supervised learning on vast amounts of text [11]. Chinese firms have become prominent in this space; DeepSeek, a Hangzhou-based company founded in 2023, released its DeepSeek-R1 model in January 2025 with performance comparable to contemporary models from OpenAI [10]. DeepSeek’s models are described as “open-weight,” meaning the exact parameters are openly shared, but the training data is not openly licensed [10]. The ChLogic benchmark provides a structured method to probe whether reasoning abilities measured in English transfer to other languages. The scientific method, which has characterized modern science since at least the 17th century, relies on hypothesis testing through experiments and statistical analysis, with the requirement that hypotheses be falsifiable [4]. By systematically varying surface realizations while holding logical structure constant, ChLogic creates a controlled test of multilingual reasoning robustness [2]. The authors describe it as a useful stress test for the robustness of multilingual reasoning [2].