ZIPP:Zero-shot Image Personalization from Personas

A new model called ZIPP can personalize AI-generated images using only a short natural-language description of a user’s tastes, eliminating the need for per-user training data or weight updates, according to research published on arXiv. [1] The system, whose name stands for zero-shot image personalization from personas, conditions image generation on concise text descriptors of a user’s identity and aesthetic sensibilities. It uses a large language model to rewrite prompts from the perspective of a given persona, steering a diffusion model toward personalized outputs without any user-specific data. [1] To build these personas at scale, the researchers trained an inductive Graph Attention Network on a Reddit interaction graph containing 22 million users. The network uses dual contrastive objectives that align graph structure with visual behavior, and a multimodal large language model then verbalizes the learned representations into natural-language personas. [1] The team also introduced ZIPBench, a benchmark for zero-shot personalization comprising 1,500 users, graph-mined personas, and 40,000 generated images. Across four benchmarks and 14 large language models from five model families, persona conditioning delivered consistent gains of 13 to 20 percent, with frontier models showing the largest improvements. [1] In few-shot settings, ZIPP matched or exceeded the performance of fine-tuned baselines that had been trained on more than 100 examples per user. The model recorded a preference distributional divergence score of 0.16 on the CMMD metric, compared with 0.55 for existing methods. [1] Human evaluators preferred ZIPP’s outputs over generic generation 79 percent of the time, and over all fine-tuned baselines between 58 and 65 percent of the time. An IPF-normalized demographic evaluation also indicated that the approach substantially reduces subpopulation bias present in prior methods. [1] Existing text-to-image models typically optimize for aggregate aesthetics rather than individual taste, and previous personalization techniques have required dense interaction histories or per-user fine-tuning, which fail in cold-start settings and collapse context-dependent preferences into a static representation. [1]