Improving Labeling Consistency with Detailed Constitutional Definitions and AI-Driven Evaluation

A new AI-driven labeling workflow can reduce cross-model inconsistency in content moderation by up to 57 times compared to traditional paragraph-length category definitions, according to research posted on arXiv [1][2]. The study addresses a persistent problem in automated labeling pipelines: simple written category definitions are not detailed enough for human annotators to produce accurate, consistent golden labels [1][2]. When definitions lack specificity, labelers fall back on intuition, causing labels to drift from the written rules and degrading both accuracy and consistency [2]. The proposed workflow uses AI to draft a per-category “constitution” that defines each label in enough detail to cover edge cases [1][2]. A frontier large language model then interprets that constitution on each input to generate the golden label [1][2]. The researchers evaluated the method on three content moderation categories—harassment, hate speech, and non-violent crime—and found that it reduced cross-model inconsistency by up to 57x compared to paragraph definitions [1][2]. A key design choice is that the human remains responsible for high-level decisions about what each category should mean, rather than making individual labeling calls [2]. Cross-model disagreement is used to diagnose specification gaps, turning inconsistencies into signals for improving the definitions [2]. The paper also introduces a dual-axis safety evaluation that scores intent and content independently over an entire conversation, allowing downstream consumers to act on either axis or both [1][2]. The findings land as regulators and platforms face mounting pressure to govern online speech at scale. The European Union adopted a common legal framework for AI with the AI Act in 2024, and jurisdictions worldwide are developing policies for promoting and regulating artificial intelligence [3]. Organizations deploying AI are expected to create trustworthy systems, adhere to established principles, and take accountability for mitigating risks [3]. Content moderation remains a high-stakes domain. Fake news websites deliberately publish hoaxes, propaganda, and disinformation, often using social media to amplify their reach for financial or political gain [5]. During the 2016 U.S. presidential election, fraudulent articles spread widely on social media, and U.S. intelligence officials said Russia was engaged in spreading fake news [5]. Computer security firm FireEye concluded that Russia used social media as part of a cyberwarfare campaign [5]. In response, Google and Facebook banned fake sites from using online advertising, and Facebook partnered with fact-checking organizations including Snopes.com, FactCheck.org, and PolitiFact [5]. Automated labeling pipelines that rely on inconsistent human annotations risk amplifying such harms. The AI-driven constitutional approach offers a path to more reliable training data by shifting human effort from per-item labeling to definitional oversight, while the frontier model handles the volume [2].