Small LLMs: Pruning vs. Training from Scratch

Pruning large language models can produce smaller, capable models more efficiently than training from scratch, according to a study that tested six methods on Meta’s Llama-3.1-8B [1]. The paper, posted to arXiv on 12 June 2026, examined pruning ratios between 0.5 and 0.8 under two token-matched settings [1]. When the same training token budget was applied, pruned initialization consistently outperformed random initialization, demonstrating that the parent model provides a strong starting point [1]. The advantage narrowed as the training token budget grew and as the pruning ratio rose, nearly vanishing at the highest ratio studied [1]. In a second setting, training from scratch was given the full token budget consumed by the entire pruning pipeline. At finer granularities, pruning still retained an advantage, while coarser structured pruning could be matched or surpassed [1]. The authors concluded that the parent model transfers knowledge that additional training tokens alone cannot fully recover, but only at fine granularity [1]. The findings arrive as the AI industry continues to debate the cost of building capable models. DeepSeek, the Chinese AI company founded in July 2023, reported training its V3 model for roughly $6 million, far less than the estimated $100 million cost of OpenAI’s GPT-4 in 2023 [7]. DeepSeek’s models are described as open-weight, meaning the exact parameters are shared but the training data is not openly licensed [7]. Meta’s Llama 3.1, the parent model used in the pruning study, was cited by DeepSeek as requiring roughly ten times the computing power of its own comparable model [7]. Large language models are defined as machine learning models with many parameters, trained with self-supervised learning on vast amounts of text [8]. Other major families include Alibaba Cloud’s Qwen, many of which are distributed under the Apache 2.0 license [9]. The pruning study’s recommendation is direct: with a large pretrained model in hand and a limited training token budget, pruning is better than training from scratch; when the training budget is not limited, training from scratch can be competitive for coarser pruning [1].