Density Field State Space Models: 1-Bit Distillation, Efficient Inference, and Knowledge Organization in Mamba-2

A new compression framework called Density Field State Space Models (DF-SSM) reduces the Mamba-2 1.3B model to a 278 MB footprint — 9.7 times smaller than its 2.7 GB FP16 teacher — while delivering a 21.4-fold inference speedup on GPU, researchers report [1]. The DF-SSM framework distills state space models into a 1-bit scaffold supplemented by an int8 low-rank correction [1]. The resulting compressed model runs downstream tasks within 2 to 4 percentage points of BitMamba-2, a 1.58-bit model trained from scratch on 150 billion tokens [1]. The distillation process itself requires only 32 million tokens and six hours on a single A100 GPU, though it depends on a pretrained FP16 teacher model [1]. To achieve the speedup, the team built an inference pipeline that combines cuBLAS INT8 tensor cores for the scaffold matrix multiplication, custom CUDA kernels for stateful SSM and convolution operations, and an AVX-512 CPU backend for deployment across both GPU and CPU environments [1]. Beyond compression, the researchers probed how knowledge is organized inside the distilled model. They identified three distinct processing phases: intent classification in layers 0 through 3, which operates in an abstract space without vocabulary alignment; knowledge retrieval concentrated in layers 25 through 35, where factual associations localize to a five-layer window; and output formatting in layers 36 through 47, where category structure dissolves [1]. Analysis of 445 factual prompts across 19 categories showed that early-layer classification is syntactic — driven by template structure — rather than semantic [1]. The model exhibited well-organized knowledge representations despite weak factual recall, a finding the authors say suggests that representational structure may precede factual strength [1]. State space models like Mamba-2 represent a departure from the transformer architectures that have dominated large language models since the transformer was first described in 2017 [3]. Deep neural networks, which underpin both families, consist of connected layers of artificial neurons that learn hierarchical representations from data, a process typically accelerated by graphics processing units [4]. The DF-SSM compression technique addresses the growing computational cost of deploying such models by drastically shrinking model size while preserving most downstream performance [1].