PixJail: Self-Evolving Paper-to-Pipeline Reproduction for Text-to-Image Jailbreak Evaluation

A new framework called PixJail aims to make the evaluation of Text-to-Image jailbreak attacks reproducible by automating the entire pipeline from research paper to benchmark, its creators report in a preprint posted to arXiv [1]. The framework, detailed in a paper submitted on June 23, 2026, is designed as a self-evolving agent that takes a T2I jailbreak paper and optional reference code, then constructs a paper-specific attack module and a runnable evaluation pipeline under a unified contract [1]. The authors argue that T2I jailbreak evaluation is not a single prompt-level test but a pipeline-level problem involving prompt transformation, image generation, safety filtering, and multimodal judging, which makes results across papers difficult to reliably reproduce and fairly compare [2]. PixJail maintains a memory bank that stores paper digests, attack evolution patterns, reusable templates, failure cases, and versioned artifacts, enabling future reproduction efforts to reuse prior experience [1]. The team reproduced eleven representative T2I jailbreak methods, including both code-available and code-unavailable papers [2]. Under their original settings, the framework recovered prior results with an average error of 2.1% and a median error of 0% [1]. The paper appears on arXiv, the open-access repository of electronic preprints that began on August 14, 1991, and now receives about 24,000 submissions per month [6]. The work is listed under the Computer Science > Cryptography and Security subject area and is accessible through arXivLabs, a framework that allows community collaborators to develop and share new features directly on the arXiv website [1][4]. arXivLabs, formalized in 2020, sets guidelines for collaborations between arXiv and third parties, ensuring that partners share arXiv’s values of openness, community, excellence, and user data privacy [4]. The PixJail preprint page includes links to code and data tools such as Hugging Face and CatalyzeX Code Finder for Papers, as well as bibliographic tools like the Bibliographic Explorer and Connected Papers [1][5]. The authors state their hope that PixJail can serve as a unified foundation for future T2I jailbreak reproduction and evaluation, significantly reducing manual effort [2].