An Empirical Analysis of Factual Errors in Human-Written Text and its Application

A new study submitted on 26 Jun 2026 finds that even high-performance large language models struggle to detect factual errors in human-written text, a task the researchers say has been neglected amid the focus on AI hallucinations [1]. The research, posted to the arXiv preprint repository, introduces a taxonomy of human-induced factual errors derived from analyzing corrections in newspaper articles [1]. The authors identified characteristic error categories, including kanji misconversions and numeral classifier errors, which are not typically represented in existing benchmarks for detecting hallucinations in machine-generated text [1]. Factual Error Detection (FED) is the task of identifying factually incorrect spans within a document [1]. To test detection capabilities, the team created synthesized realistic test cases and evaluated several large language models [1]. GPT-5.4, a successor to OpenAI's GPT-4, achieved a word-level F1 score of 52% on the synthetic evaluation data, a result the authors describe as highlighting the difficulty of the task [1]. GPT-4 was originally released in March 2023 and preceded the GPT-5 series [4]. Language model benchmarks typically provide standardized datasets and metrics to measure performance on tasks such as text classification and reasoning [5]. The study was submitted to arXiv, an open-access repository for electronic preprints that has hosted scientific papers since 1991 and now receives about 24,000 submissions per month [9]. The paper appears with a suite of community-developed tools under the arXivLabs framework, which allows third-party collaborators to build experimental features on the platform while adhering to arXiv's values of openness, community, and user data privacy [8]. These tools include the Bibliographic Explorer for navigating citation trees and the CORE Recommender for discovering related open-access papers [7][8]. The researchers argue that the detection of factual errors in human-written text has been relatively neglected as attention shifted toward hallucinations in large language model outputs [1]. Their detailed analysis by detection difficulty aims to establish a clearer picture of the current state of FED [1].