SketchSong: Hierarchical Song Generation with Sketch Planning and Fine-Grained Multi-Track Modeling

Researchers have proposed two new song generation frameworks, SketchSong and SegTune, to improve upon existing systems that struggle with arrangement planning and musical part modeling.

SketchSong, presented in a paper submitted on June 2, 2026^[1], addresses the limitations of current song generation systems through hierarchical song generation with sketch planning and fine-grained multi-track modeling. The framework first predicts a compact sequence of high-level sketch tokens derived from compressed audio representations, and then generates audio tokens conditioned on these sketches, giving it an explicit arrangement plan before detailed audio generation. SketchSong explicitly models four tracks: vocals, bass, drums, and other instruments, capturing the roles and interactions of different musical parts more precisely. Meanwhile, SegTune, submitted on May 31, 2026^[2], uses a Diffusion Transformer-based framework to enable fine-grained controllability, allowing users or large language models (LLMs) to specify local musical descriptions aligned to song segments. SegTune also includes an LLM-based duration predictor for precise lyric-to-music alignment.