You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations

A new training-free quantization framework called Task-Aware Quantization (TAQ) allocates precision to transformer layers based on a target task, rather than treating all layers uniformly, according to a preprint posted to arXiv [1]. The method, detailed in a paper revised in June 2026, addresses a limitation in standard post-training quantization (PTQ) for large language models. Many LLM applications require only narrow capabilities, yet conventional PTQ methods allocate precision without considering the target task, potentially wasting bits on less relevant layers while over-compressing critical ones [1]. TAQ uses a small set of unlabeled task calibration prompts to assign higher precision to task-relevant layers under a fixed bit budget [1]. The framework estimates layer importance from hidden representations and output sensitivity. The authors instantiate it with three scoring rules: TAQ-IS, based on activation information and stability; TAQ-KL, based on output-distribution sensitivity under a quantization-noise proxy; and TAQ-O, a label-informed oracle diagnostic for analyzing layer sensitivity [1]. Across several benchmarks, TAQ outperformed task-agnostic baselines in most settings, with especially strong gains in the accuracy–memory ratio [1]. The paper further validates that these gains translate to real deployment behavior through hardware throughput and latency measurements, and analyzes calibration robustness and residual-stream error propagation [1]. The submission history shows the manuscript was first uploaded on November 9, 2025, at a size of 180 KB, with subsequent revisions growing to 5,525 KB by version four [1]. The work was led by Raz Lapid, and a reference implementation has been made available through an anonymous repository [1]. The research was conducted within the arXivLabs framework, which allows collaborators to develop and share new arXiv features directly on the platform under values of openness, community, excellence, and user data privacy [1].