A Study on Question-Answer Dataset for LLM Safety Evaluation with a Focus on Illegal Activities

Researchers have released a study detailing a question-answer dataset designed to evaluate the safety of large language models (LLMs) when confronted with prompts about illegal activities, according to a paper published on arXiv [1]. The paper, submitted on 28 May 2026, introduces methods for creating question-answer examples and a rubric for evaluating LLM-generated responses [1]. The work is based on a manual analysis of an existing dataset called AnswerCarefully [1]. The study's authors state that the outcomes are intended to be shared with the "JAI-Trust" project [1]. The research addresses a specific vulnerability in AI systems, where models might be manipulated into generating harmful content. This focus on safety evaluation comes as the broader field of artificial intelligence grapples with the challenge of making systems transparent and accountable. Explainable AI (XAI) is a field of research that explores methods providing humans with intellectual oversight over AI algorithms, countering the "black box" tendency of machine learning where even designers cannot explain a specific decision [5]. XAI aims to help users assess safety and scrutinize automated decision-making in applications [5]. The development of robust safety evaluations also unfolds against a backdrop of rapid AI expansion in India, where the market is projected to reach $8 billion by 2025, growing at a compound annual growth rate of 40% from 2020 [3]. This growth has been bolstered by government initiatives like NITI Aayog's 2018 National Strategy for Artificial Intelligence [3]. However, the report notes that the growth of AI in India has also led to an increase in the number of cyberattacks that use AI to target organizations [3]. The safety concerns addressed by the new dataset are distinct from the theoretical risks associated with artificial general intelligence (AGI), a hypothetical type of AI that would match or surpass human capabilities across virtually all cognitive tasks [4]. While a 2020 survey identified 72 active AGI research and development projects across 37 countries, contention exists over whether AGI represents an existential risk, with some experts stating that mitigating the risk of human extinction posed by AGI should be a global priority [4]. The current study remains focused on the immediate, practical dangers of existing narrow AI systems generating content related to illegal activities [1].