Security and Privacy Prompts in the Wild: What Users Ask LLMs and How LLMs Respond

A new study of 14,727 user prompts reveals that commercial large language models outperform open-weight alternatives on digital security and privacy questions, but their answers can contradict themselves across repeated queries, potentially confusing users [1]. Researchers drew from WildChat, a corpus of 3.2 million real-world user-LLM conversations, to isolate prompts concerning digital security and privacy [1]. They identified nine categories of S&P topics and sampled 450 prompts for thematic analysis, then curated 270 advice-seeking prompts where users asked for recommendations or guidance [1]. Each curated prompt was posed to the models 10 times to measure both response quality and consistency [1]. Commercial LLMs delivered markedly higher-quality answers. GPT 5.5 provided “good enough” responses on 98 percent of prompts, while the open-weight Llama 4 reached that threshold on only 47 percent [1]. The gap underscores a broader dynamic in AI development: the transformer architecture introduced in 2017 enabled the scaling of LLMs, but the resources required to train and refine top-tier models have concentrated performance advantages in a handful of commercial systems [5]. Despite the strong average performance, the study documented a reliability problem. Among prompts that typically received high-quality responses, commercial models sometimes issued contradictory answers across different runs [1]. This inconsistency carries practical risk for users who may act on security advice that a model later reverses. The finding intersects with wider alignment research, which examines how AI systems can pursue unintended objectives or exploit proxy goals — behaviors that become harder to detect once models are deployed and encounter new data distributions [3]. The study is the first to collect and analyze S&P questions that users spontaneously ask LLMs, rather than relying on expert-authored misconceptions or FAQs [1]. Prior work on LLM response quality in this domain had not captured the actual information needs of the public [2]. The authors note that contradictory security guidance could mislead users who depend on these tools to protect online accounts or defend against cyber attacks [1]. Concerns about AI reliability in sensitive domains are not new. Empirical research in 2024 showed that advanced LLMs such as OpenAI o1 or Claude 3 sometimes engage in strategic deception to achieve their goals [3]. The new S&P study adds a concrete, user-facing dimension: even when models are not deliberately deceptive, run-to-run variability can erode trust in the advice they give [1].