Rational Sparse Autoencoder

Researchers have introduced the Rational Sparse Autoencoder (RSAE), a new architecture that replaces the fixed activation functions in standard sparse autoencoders with trainable rational functions, according to a paper submitted on 12 Jun 2026 [1]. Sparse autoencoders (SAEs) are widely used in mechanistic interpretability, a field that seeks to understand the internal computations of deep neural networks [3]. Current SAE designs rely on fixed encoder nonlinearities, such as ReLU, JumpReLU, and TopK, which hard-code a specific sparsity mechanism into the model [1]. The authors of the new paper argue this constraint can distort the trade-off between reconstruction fidelity and sparsity [1]. The RSAE addresses this by using a trainable rational function as the encoder activation, providing a more flexible function class that can adapt to the geometry of pre-activation values [2]. The RSAE’s rational activation is capable of uniformly approximating the activation primitives of existing SAE families on compact domains [2]. For the TopK baseline, this approximation uses a thresholded gate obtained after a separating top-k threshold is supplied [2]. The implementation follows a two-stage pipeline: an initialisation procedure copies the weights from a pre-trained baseline SAE and fits rational coefficients using a relaxed Remez exchange on synthetic data, followed by a fine-tuning step under a standard sparsity-regularised reconstruction objective [2]. Empirical tests were conducted on residual-stream activations from three open-weight language models [2]. Across all three baseline activation families, the RSAE showed strict improvement after fine-tuning on both reconstruction-side metrics and downstream-behaviour metrics, without sacrificing feature-level interpretability as measured by sparse probing [2]. These gains held consistent across host language models, baseline activation families, and the full range of tested sparsity levels [2]. The upgrade adds only a handful of scalar parameters per autoencoder and runs in minutes on a single consumer GPU [2].