Pigeonholing: Bad prompts hurt models to collapse and make mistakes

A new study identifies a phenomenon called “pigeonholing,” where bad conversational contexts cause large language models to suffer performance drops and mode collapse, even without malicious intent [1]. The research, posted to arXiv on June 23, examines how unintentionally bad prompts degrade model outputs across 10 verifiable and open-ended tasks using 10 different models [1]. The authors define pigeonholing as a collapse triggered when a user suggests an incorrect solution or when a conversation history includes the assistant’s own previous mistakes [2]. Experiments show the degradation manifests in three ways: models repeat incorrect answers from context, leading to a 38-40% performance drop; they converge on a narrow set of responses in coding and text generation without exploring alternatives; and they flip their stance on controversial topics to align with the user or the assistant’s earlier claims [1]. The effect worsens with conversation length. As repeated mistakes increase from 1 to 5, performance drops by an additional 14+% per turn [2]. Large language models are machine learning systems with many parameters, trained on vast amounts of text through self-supervised learning [7]. The study notes that pigeonholing-induced mode collapse can occur even when the provided example is correct [2]. To address the problem, the researchers propose a method called RLVR with synthetic errors. This approach improved model performance by 43-60% under bad contexts compared to vanilla RLVR baselines [1]. The work arrives as LLM deployment accelerates across industry. Chinese firm DeepSeek, founded in July 2023, launched its R1 model in January 2025 with performance comparable to OpenAI’s GPT-4 and o1, while claiming training costs of US$6 million — far below the US$100 million reported for GPT-4 in 2023 [6]. The arXiv paper is accessible through the platform’s integration with Hugging Face Spaces, a collaboration that allows researchers and the community to attach interactive demos directly to preprint pages [3][4]. Users can add a Space to an arXiv paper by including the paper’s link in the Space README file or by linking an intermediate model hosted on the Hugging Face Hub [5]. The demo infrastructure, built with open-source tools such as Gradio and Streamlit, aims to increase reproducibility and let a wider audience inspect model behavior without writing code [3].