Code as a Weapon: A Consensus-Labeled Prompt Bank for Measuring Coding-Model Compliance with Malicious-Code Requests

A new consensus-labeled prompt bank aims to resolve a critical gap in AI safety testing by distinguishing requests for executable malicious code from requests for harmful security knowledge, providing a standardized instrument to measure whether coding-specialized models meet the stricter refusal standards their outputs demand [1]. The field currently cannot determine if coding models clear a higher refusal bar than general-purpose chat models, despite the heightened risk that a compliant coding model can return a working weapon such as a keylogger or ransomware stub [1]. Existing refusal benchmarks for malicious code are fragmented, mixing requests for ready-to-run software with requests for information a human must still operationalize, and reporting refusal rates over non-comparable corpora [1]. This inconsistency has left no single statistic to measure the property that matters most: whether a model will refuse to generate executable threats [1]. The new prompt bank consolidates eight existing corpora—including ASTRA, CySecBench, AdvBench/harmful_behaviors, JailbreakBench, MalwareBench, RedCode, RMCBench, and Scam2Prompt—and classifies them under a five-judge consensus protocol [1]. The panel evaluated 6,675 prompts, yielding 33,375 individual judgments [1]. The judges achieved a Fleiss' kappa of 0.767, with a 95% confidence interval of 0.755 to 0.777, indicating substantial agreement [1]. In 95.0% of prompts, at least four judges agreed, and 76.9% of prompts received unanimous agreement [1]. The panel also reproduced an earlier four-corpus release with a Cohen's kappa of 0.952 on 3,133 shared prompts [1]. The released bank comprises 4,748 consensus-CODE prompts, which request executable malicious code, and 1,923 consensus-KNOWLEDGE prompts, which request harmful security information [1]. This distinction is central to the instrument's purpose: a coding model that complies with a malicious request can return code that runs as written, an asymmetry in severity that implies coding-specialized models should clear a higher refusal bar than general-purpose chat models [1]. The bank provides a reliability-quantified basis for testing whether coding models meet that stricter standard [1]. The broader ethical stakes of AI systems that generate harmful outputs extend beyond code generation. The ethics of artificial intelligence encompasses algorithmic biases, fairness, accountability, transparency, and regulation, particularly where systems influence or automate human decision-making [3]. Application areas such as healthcare, education, criminal justice, and the military carry particularly important ethical implications [3]. The prompt bank's focus on executable malicious code intersects with these concerns by addressing a concrete pathway through which AI systems could cause direct harm if safety measures fail.