MORTAR: Multi-turn Metamorphic Testing for LLM-based Dialogue Systems

A team led by Guoxiang Guo has proposed MORTAR, a metamorphic testing framework designed to address the test oracle problem in multi-turn LLM-based dialogue systems, according to a paper posted on arXiv [1]. The approach, detailed in a preprint first submitted in December 2024 and last revised in June 2026, formalizes multi-turn dialogue testing by automating the generation of question-answer test cases with multiple dialogue-level perturbations and metamorphic relations [1][2]. The automated metamorphic relation matching mechanism is designed to operate without relying on LLM judges, a feature the authors argue increases flexibility and efficiency [2]. Large language models, which are trained on vast amounts of text using self-supervised learning, underpin a growing number of conversational agents [8]. As these dialogue systems are deployed in daily life, quality assurance has become more critical, yet testing methods for multi-turn interactions—the common real-world usage pattern—remain underexplored due to the difficulty of defining expected outputs across extended exchanges [2]. In experiments covering six popular LLM-based dialogue systems, MORTAR revealed over 150% more bugs per test case compared to a single-turn metamorphic testing baseline [1][2]. The bugs identified were also rated higher in diversity, precision, and uniqueness [2]. The paper appears on arXiv, an open-access repository that hosts preprints across physics, computer science, and related fields and has grown to receive roughly 24,000 submissions per month as of late 2024 [6]. The preprint’s abstract page includes links to community-developed tools under the arXivLabs framework, such as the Bibliographic Explorer and CORE Recommender, which provide citation navigation and open-access paper recommendations [4][5]. arXivLabs, launched in 2020, allows third-party collaborators to build experimental features on top of the repository while adhering to user privacy and openness principles [4]. The MORTAR paper’s revision history shows four versions, with the most recent uploaded on June 17, 2026 [1].