IGLU: The Integrated Gaussian Linear Unit Activation Function

A new activation function called IGLU, designed to improve gradient flow in deep neural networks, has been introduced by researchers. The parametric function uses a heavy-tailed Cauchy gate to provide non-zero gradients for all finite inputs, addressing a known limitation of widely used alternatives like GELU [1]. The work, detailed in a paper posted to the arXiv preprint repository, derives IGLU as a scale mixture of GELU gates under a half-normal mixing distribution [1]. This mathematical formulation yields a closed-form expression whose gating component is exactly the Cauchy cumulative distribution function, creating a one-parameter family that continuously interpolates between identity-like and ReLU-like behavior via a single sharpness parameter [1]. Unlike the Gaussian gate used in GELU, IGLU's heavy-tailed Cauchy gate decays polynomially in the negative tail, guaranteeing non-zero gradients for all finite inputs and offering greater robustness to vanishing gradients [1]. The paper was submitted by researcher Mingi Kang and revised on June 12, 2026 [1]. arXiv, which began on August 14, 1991, now receives about 24,000 article submissions per month and hosts over two million papers, serving as a primary distribution channel for preprints in computer science and related fields [6]. The repository is not peer-reviewed, with papers approved after moderation [6]. The research team also introduced IGLU-Approx, a computationally efficient rational approximation of IGLU expressed entirely in terms of ReLU operations that eliminates transcendental function evaluation [1]. Evaluations were conducted on CIFAR-10, CIFAR-100, and WikiText-103 datasets across ResNet-20, ViT-Tiny, and GPT-2 Small architectures [1]. IGLU achieved competitive or superior performance on both vision and language datasets against ReLU and GELU baselines, with IGLU-Approx recovering this performance at substantially reduced computational cost [1]. The paper notes that employing a heavy-tailed gate leads to considerable performance gains in heavily imbalanced classification datasets [1]. The initial submission on March 6, 2026, was 453 KB, and the revised version was 455 KB [1].