LLM Compression with Jointly Optimizing Architectural and Quantization choices

Researchers have proposed a framework for compressing large language models (LLMs) that achieves faster inference and higher accuracy than existing methods.

Deploying LLMs is challenging due to their significant memory and computational requirements^[1]. Compressing pre-trained LLMs for edge devices is a compelling alternative to developing small models from scratch. A proposed differentiable NAS framework jointly optimizes architectural configurations and mixed-precision quantization for LLMs, achieving up to 1.4x faster inference than sequential NAS-then-quantization baselines at comparable accuracy^[1]. The framework also achieves up to 6% higher average accuracy across seven reasoning tasks at equivalent latency. Another study found that the proposed method reduces WikiText perplexity by up to 21% compared to state-of-the-art weight-activation quantization baselines^[2]. It also achieves up to 59% and 85% lower perplexity on WikiText and C4, respectively, compared to leading weight-only quantization methods. The method delivers superior perplexity and reasoning performance at ultra-low bits compared to state-of-the-art joint pruning-and-quantization techniques^[2].