A large-scale pipeline for LLM-assisted corpus annotation: variation and change in the English consider construction

A research team has detailed a large-language-model pipeline that automates grammatical annotation across massive text corpora, completing a task on 143,933 concordance lines in under 60 hours while maintaining accuracy above 98 percent, according to a paper posted on arXiv [1]. The method, developed by Cameron Morin and colleagues, addresses what the authors describe as a “significant methodological bottleneck” in corpus linguistics, where the rapid expansion of natural-language datasets has outpaced the capacity for manual labeling [1][2]. The pipeline is structured around four stages: prompt engineering, pre-hoc evaluation, automated batch processing, and post-hoc validation [1]. It was tested on the English evaluative “consider” construction — patterns such as “consider X as,” “consider X to be,” and “consider X Ø Y” — using data drawn from the Corpus of Historical American English [1]. Processing was conducted through the OpenAI application programming interface [1][2]. OpenAI, the San Francisco-based organization behind the GPT family of large language models, has been a central force in the recent acceleration of generative AI development [4]. The broader AI boom of the 2020s has seen large language models applied to tasks ranging from protein-folding prediction to text generation, with ChatGPT becoming the fourth-most-visited website globally by 2025 [5]. The annotation run processed 143,933 “consider” concordance lines in under 60 hours and achieved accuracy exceeding 98 percent on two distinct annotation procedures [1][2]. A Bayesian multinomial generalized additive model was then fitted to 44,527 true positives of the evaluative construction, revealing genre-specific trajectories of change that had not been previously documented [1]. The findings allowed the researchers to propose new hypotheses about how register formality interacts with competing pressures of morphosyntactic reduction and enhancement [1][2]. High-quality labeled datasets have long been difficult and expensive to produce because of the time required for manual annotation, a constraint that has shaped progress in supervised machine learning [3]. The paper argues that large language models can now perform a range of data-preparation tasks at scale with minimal human intervention, opening research questions that were previously beyond practical reach [1][2]. The authors also note that implementation requires attention to costs, licensing, and ethical considerations [1]. The preprint was posted on arXiv, the open-access repository that hosts scientific papers across disciplines including computer science and linguistics and that, as of late 2024, was receiving approximately 24,000 submissions per month [6]. The version cited here is the third revision, dated June 2026 [2].