Can Large Language Models Reliably Code Qualitative Humanitarian Data? A Benchmark Study Against Human Expert Adjudication

A benchmark study of 46 large language models finds that several can code qualitative humanitarian data with reliability comparable to experienced human coders, though the authors caution that aggregate scores alone are insufficient for deployment decisions [1]. The study, submitted in 2026, evaluated model performance against a human Gold Standard using 150 high-fidelity synthetic humanitarian transcripts [1]. Researchers combined inter-rater reliability testing with Krippendorff's alpha, discrepancy analysis, and qualitative assessment across humanitarian-specific criteria [1]. The findings indicate that multiple LLMs can perform deductive coding at reliability levels comparable to experienced human coders, particularly when structured prompts and reasoning-enabled configurations are used [1]. However, the paper warns that aggregate reliability metrics mask important variation. Models differed in their ability to recognize needs expressed indirectly, needs falling outside predefined categories, and protection-relevant concerns such as physical safety and discrimination [1]. The authors recommend that appropriate deployment requires structured codebooks, reasoning-enabled models, attention to theme-specific performance, and tiered oversight focused on categories where miscoding carries the greatest programmatic consequences [1]. Large language models are machine learning systems with many parameters, trained with self-supervised learning on vast amounts of text [7]. The study's focus on open-weights models deployed on self-hosted infrastructure reflects a broader trend toward making AI tools accessible without reliance on cloud services [1]. This approach echoes developments in other generative AI domains, such as the release of Stable Diffusion in 2022, whose code and model weights were made publicly available and could run on consumer hardware with as little as 2.4 GB of VRAM, marking a departure from proprietary models accessible only via cloud services [8]. The humanitarian sector's exploration of LLMs comes amid rapid evolution in the broader AI landscape. Chinese firm DeepSeek, founded in July 2023, demonstrated that competitive models could be trained at significantly lower cost, claiming its V3 model was trained for US$6 million compared to the US$100 million reported for OpenAI's GPT-4 in 2023 [6]. DeepSeek's models are described as open-weight, meaning the exact parameters are openly shared, though the training data is not openly licensed [6]. For sensitive humanitarian data, the study suggests that open-weights models deployed on self-hosted infrastructure may offer a viable path for combining analytical scalability with stronger data governance [1]. The authors stress that LLMs can materially expand humanitarian analytical capacity, but not as substitutes for human judgment [1].