Do Encoders Suffice? A Systematic Comparison of Encoder and Decoder Safety Judges for LLM Adversarial Evaluation

A new study examines whether fine-tuned encoder classifiers can match the accuracy of large language model judges in detecting harmful chatbot outputs, while offering lower cost and latency for production safety systems [1]. The paper, submitted on 24 Jun 2026, evaluates classifiers from the ModernBERT family — including ModernBERT and Ettin — against a suite of LLM-based safety judges and rule-based baselines [1]. The work addresses a growing operational tension: LLM-based judges are effective but often too slow and expensive to deploy at the scale required by consumer-facing applications [2]. The researchers benchmarked encoder classifiers against rule-based prefix matching, fine-tuned LLM classifiers, and LLM judges employing methodologies from StrongReject, ShieldGemma, JailbreakBench, AILuminate, SorryBench, and a Claude-as-a-judge setup [2]. Fine-tuned safety classifiers such as LlamaGuard 3 and LlamaGuard 4 were also included in the comparison [2]. Encoder classifiers were fine-tuned on judge-labeled data using a majority-voting label strategy and then evaluated on a gold-standard holdout dataset [2]. Performance was reported using F1 score, false negative rate, and precision-recall metrics [2]. The analysis further breaks down results by attack technique — single-turn prompting, decomposition, escalation, and context manipulation — to identify where encoder classifiers align with or diverge from LLM-based judges [2]. The study’s findings aim to provide guidance on when encoder classifiers can serve as cost- and latency-efficient alternatives to LLM-based safety evaluation [2]. The research comes as organizations deploying LLMs in chatbots and everyday applications face mounting pressure to implement guardrails that balance effectiveness with operational feasibility [2]. While the primary research focuses on adversarial safety evaluation, the broader challenge of building reliable, resource-efficient classifiers spans multiple domains. For instance, transfer-learning strategies that leverage larger datasets to improve performance on smaller, specialized datasets have been explored in catalysis informatics, where models trained on the OC20 dataset were used to boost results on OC22 [4]. Similarly, the concept of fine-tuning pre-trained models for specific downstream tasks has parallels in molecular biology, where transcription factors — proteins that regulate gene expression — must bind specific DNA sequences with high precision to control cellular processes [7]. These cross-domain examples underscore the general importance of developing classifiers that are both accurate and computationally practical. The paper’s systematic comparison offers a framework for practitioners deciding whether to deploy encoder-based safety classifiers in production environments, where latency and cost constraints often rule out the use of full LLM judges for every inference call [1][2].