An Expanded Synthetic Conversation Dataset for Multi-Turn Smishing Detection

Researchers have released COVA-X, an expanded synthetic dataset of 10,985 multi-turn conversations designed to improve smishing detection, and report that a transformer model outperformed a traditional machine-learning baseline for the first time on the task. The dataset, detailed in a paper posted to arXiv, expands the original COVA corpus of 3,201 conversations by a factor of 3.43 and spans eight scam categories targeting older adults [1][6]. The prior work had found that XGBoost with TF-IDF features achieved 72.5% accuracy and a 0.691 macro F1 score, while transformer models lagged behind due to input truncation and limited training data [1][2]. The new COVA-X corpus was built with an improved generation pipeline that the authors say addresses contamination, label mismatch, stage-direction bleed, and prompt-design failures identified in the first iteration [1][6]. After retraining all classifiers on the expanded dataset, the Longformer model reached 79.71% accuracy and a 0.7786 macro F1 score, compared with 78.43% accuracy and 0.7563 macro F1 for XGBoost [1][6]. The paper states that this result confirms the hypothesis that transformer models require larger conversational corpora to realize their contextual advantages [2][6]. The strongest gains were observed for transformer-based architectures, indicating that contextual representations become more effective as dataset scale and quality improve [6]. The authors also document a quality lifecycle analysis. The label correction rate improved by a factor of 12.7, dropping from an initial 49.8% to a final 3.9% [1][2]. An architectural intervention reduced virtual-kidnapping artifact rates from 67.1% to 46.5% [1][2]. A pre- and post-cleanup sensitivity analysis showed that dataset refinement recovered genuine label-relevant signal across all three classifier architectures tested [2][6]. Synthetic datasets for scam detection have been explored in other recent work. One approach used ChatGPT-4 to generate 90 seed conversations across four scam categories—authority, job, love, and investment—and then leveraged GPT-3.5 Turbo to produce 10 variants per seed, with manual vetting to discard low-quality outputs [7]. A separate synthetic multi-turn scam and non-scam phone conversation dataset hosted on Hugging Face includes categories such as social security number scams, refund scams, technical support scams, and reward scams, alongside legitimate interactions like delivery confirmations and insurance calls [5]. The authors of that dataset note that synthetically generated conversations may not capture all the nuances of real-world phone interactions and that biases may be present in the generated dialogues [5]. The COVA-X paper argues that transparent documentation of iterative pipeline challenges constitutes a methodological contribution, offering a practical roadmap for future synthetic dataset construction efforts [6]. The dataset and its companion data sheet, which includes version history, per-scam-type counts, and known limitations, have been released alongside the paper [6].