Text-Only Data Synthesis for Vision Language Model Training

A new framework proposes synthesizing multimodal training data for vision-language models entirely from text, eliminating the need for real images. The approach, detailed in a paper by Xiaomin Yu and colleagues, generates two synthetic datasets through a three-stage process [1]. The framework, described on arXiv, constructs a dataset of 1.2 million semantically diverse captions by using large language models to expand sparse caption seeds [2]. A subset of 471,000 captions is then processed into multi-turn instruction-tuning tasks to support complex reasoning [2]. In a final stage, the textual caption representations are transformed into visual representations, yielding synthetic image data [2]. The resulting datasets, named Unicorn-1.2M and Unicorn-471K-Instruction, are designed for pretraining and instruction-tuning, respectively [1]. The authors state the method offers a cost-effective and scalable solution for training vision-language models by removing the dependency on real images while maintaining data quality and diversity [2]. Training vision-language models has traditionally relied on large-scale, high-quality image-text pairs, which are expensive to collect or synthesize [2]. Text data, by contrast, is abundant and inexpensive [2]. This disparity has driven research into text-only data synthesis. The proposed framework's third stage, Modality Representation Transfer, is a key differentiator, converting text-based representations into visual ones without generating actual pixel-based images [2]. Vision-language models are a class of multimodal large language models, exemplified by systems like Google DeepMind's Gemini family, which was announced in December 2023 [3]. These models typically integrate text and image understanding. The broader field of text-to-image generation has seen rapid advancement since the mid-2010s, with models such as DALL-E 2, Stable Diffusion, and Midjourney achieving outputs that approach the quality of real photographs by 2022 [4]. These systems commonly use latent diffusion models, which operate in a compressed latent space and often combine a text encoder with a diffusion-based image generator [4][5]. Diffusion models themselves learn to reverse a process of adding noise to data and have become dominant in computer vision tasks including image generation and video generation [5]. The new text-only synthesis framework represents a departure from these image-dependent pipelines by operating purely in the textual and representational domains.