Scaling Properties of Continuous Diffusion Spoken Language Models

A team of researchers has identified scaling laws for continuous diffusion spoken language models, finding that larger compute budgets allow near-optimal performance across a wider range of model and dataset sizes, according to a paper from Apple [1][2]. The study, authored by Jason Ramapuram, Eeshan Gunesh Dhekane, Amitis Shidani, and colleagues, examines whether continuous diffusion (CD) spoken language models offer a more viable path than discrete autoregressive approaches, which require significant computational and data resources to match text-based models [1][2]. The researchers introduced a metric called phoneme Jensen-Shannon divergence (pJSD) to quantify linguistic quality [2]. Their analysis shows that CD SLMs exhibit scaling laws for both validation loss and pJSD, mirroring behavior seen in autoregressive models [1][2]. A key finding involves the optimal ratio of tokens to parameters. As compute scales up, this ratio decreases, but the loss function becomes increasingly insensitive to the specific choice of data and model sizes [1][2]. The curvature of isoFLOP curves at their optima flattens, corresponding to roughly two orders of magnitude expansion in the range of model and dataset sizes that yield a loss within epsilon of the optimum [1]. This flattening opens what the authors describe as an efficient inference frontier, suggesting potential for fast inference [1][2]. When scaled to 16 billion parameters and trained on tens of millions of hours of conversational data, the models generated emotive, prosodic, multi-speaker, and multilingual speech [1][2]. However, achieving long-form coherence remains a significant challenge [1][2]. Speech perception research, which studies how human listeners recognize speech sounds and extract acoustic cues and phonetic information, provides a backdrop for evaluating such models [3]. The work builds on deep learning architectures, which use multilayered neural networks trained to process data and have been applied to speech recognition and natural language processing [4]. The paper appears on Apple's machine learning research site [1]. The broader field of spoken language modeling has seen contributions from labs such as Google DeepMind, which has developed neural network models for game-playing, protein folding, and generative AI tools including large language models [8].