Rapid Poison: Practical Poisoning Attacks Against the Rapid Response Framework

A security framework used in production AI systems, including Anthropic's ASL-3 safeguards, contains a vulnerability that allows attackers to poison its training data through prompt injection, according to new research published on arXiv [1]. The Rapid Response (RR) framework is designed to continuously improve classifiers that detect jailbreak attempts—prompts that bypass an AI model's safety restrictions. When new jailbreaks emerge, the framework generates synthetic variants for training, helping the model generalize and adapt quickly [1]. Researchers have now demonstrated that this same pipeline can be exploited. By using prompt injection, adversaries can deliver poisoned samples into the classifier's training set [1]. The paper details two distinct attack objectives. The first is a targeted poisoning attack that creates false positives, causing harmless samples to be incorrectly flagged as jailbreaks when they contain a specific feature such as a keyword, subject, or formatting style [1]. The second is a concept-based backdoor attack that induces false negatives on actual jailbreak inputs when a trigger is present, allowing malicious prompts to evade detection even when the classifier was explicitly trained against those attack strategies [1]. The researchers operated under a constrained threat model: adversaries could modify only jailbreak samples, not benign data or labels. This restriction made the second objective particularly difficult and had not been explored in prior work [1]. To overcome it, the team developed what they call the Omission Attack. This technique exploits a phenomenon in which training on unsafe samples that lack a particular concept causes the classifier to misassociate that concept's presence with the safe label [1]. The results were severe. Both attacks caused substantial, and in some cases near-complete, label flipping at a poisoning rate of only 1% [1]. The false positive rate reached up to 100%, while the false negative rate climbed as high as 96% [1]. The findings arrive amid broader concerns about the pace of AI safety research. AI safety as a field focuses on preventing accidents, misuse, and harmful consequences from artificial intelligence systems, encompassing alignment, monitoring, and robustness [3]. Researchers have expressed concern that safety measures are not keeping pace with rapid AI development [3]. The vulnerability identified in the Rapid Response framework underscores those concerns, as it targets a system specifically built to strengthen defenses against adversarial inputs [1].