Moderating Illicit Online Image Promotion for Unsafe User-Generated Content Games Using Large Vision-Language Models

A research team has built an automated system to detect illicit online promotions for user-generated content games that feature explicit or violent material, targeting a growing safety risk for children and adolescents on social media. The system, called UGCG-Guard, was developed to flag image-based advertisements that game creators use to draw players to unsafe user-generated content games, or UGCGs [1]. These platforms are increasingly popular among young users for social interaction and creative play, but they also carry a heightened risk of exposure to sexually explicit and violent content [2]. Despite those risks, few studies had examined how illicit promotions for such games spread on social media [3]. Keyan Guo and collaborators collected a real-world dataset of 2,924 images displaying diverse explicit and violent promotional material [1]. The dataset underpins UGCG-Guard, which relies on large vision-language models and a conditional prompting strategy for zero-shot domain adaptation [1]. The approach also uses chain-of-thought reasoning to interpret the context of depicted activities, allowing the model to recognize nuanced patterns specific to UGCG promotions without requiring a large labeled training set [3]. In testing, UGCG-Guard achieved a 94 percent accuracy rate in identifying images used for illicit UGCG promotion in real-world scenarios [4]. The work was presented at the 33rd USENIX Security Symposium in August 2024 [4]. Broader research on illicit content detection underscores the challenge of adapting moderation tools to new platforms and content types. A separate study found that in-context learning can generalize to entirely new illicit categories with a performance drop of less than 6 percent for more than half of evaluated categories, and it surfaced eight previously undocumented illicit categories, including usury and illegal immigration [5]. That framework, deployed on 200,000 real-world samples from search engines and Twitter without platform-specific adaptation, reached 92.6 percent accuracy, with 61.8 percent of its uniquely flagged samples corresponding to borderline or obfuscated promotional content missed by existing detectors [5]. Other recent work has explored combining vision-language models with segmentation tools to localize malicious image regions with pixel-level masks, hardening detection pipelines against adaptive attacks that target any single segmentation method [7]. These parallel efforts highlight a shift toward zero-shot and multimodal techniques that can keep pace with rapidly evolving illicit promotion tactics online.