GEMS: Geometric Constraints Enable Multi-Semantic Superposition in LLMs

Researchers have proposed GEMS, a training-free method that uses geometric constraints to enable multi-semantic superposition in large language models, preventing the model collapse that occurs when multiple behavioral directions are injected without safeguards [1]. The work, posted to the arXiv preprint repository on June 18, 2026, addresses a core limitation of activation steering, a technique that modifies a model's intermediate hidden states at inference time without retraining [1]. Existing steering methods can inject only a single semantic direction; when multiple directions are combined without constraints, model performance collapses [2]. The authors decompose this collapse into two independent sources: distributional deviation, where additive perturbations accumulate in norm across layers and push activations outside the training distribution, and directional interference, where non-orthogonal semantic vectors mutually dampen when superposed [2]. GEMS maps each failure source to a corresponding geometric constraint. It applies norm-preserving weighted superposition and targeted attention-pathway injection to counter distributional deviation, and uses real-time orthogonalization to address directional interference [2]. On the GSM8K mathematical reasoning benchmark, injecting three concurrent non-mathematical directions preserved accuracy at 98%, compared to a baseline of 92% [2]. Unconstrained addition of the same directions caused accuracy to collapse to 4% [2]. On the Wikitext-2 language modeling dataset, the same three-direction injection produced only a 2.2% increase in perplexity [2]. Component ablation studies isolated the causal role of each constraint, and layer-level probes confirmed that orthogonalized signals survive the feed-forward network pathway and reach the output distribution with semantic specificity [2]. The qualitative steering effects transferred across model architectures ranging from 3 billion to 31 billion parameters [2]. Large language models are trained with self-supervised learning on vast amounts of text and are designed for natural language processing tasks such as language generation [8]. The arXiv repository, where the paper appeared, hosts open-access preprints that are moderated but not peer-reviewed, and as of November 2024 receives about 24,000 submissions per month [6].