Your UnEmbedding Matrix is Secretly a Feature Lens for Text Embeddings

A team of researchers has identified a mechanism that degrades the performance of large language models when used as text embedding tools and proposed a linear transformation called EmbedFilter to correct it, according to a paper submitted June 5, 2026. [1] Large language models, or LLMs, show strong zero-shot abilities across many tasks but deliver suboptimal results when deployed directly as embedding models on large-scale text embedding benchmarks. [1] The authors of the paper trace this shortfall to an unexpected behavior: text embeddings, when projected onto the vocabulary space, align with frequent but semantically weak tokens. [1] That alignment suppresses the model's capacity to capture nuanced meaning. [1] The investigation focused on the unembedding matrix inside LLMs. The researchers found that this matrix encodes a latent subspace they call the "edge spectrum" space, spanned by the right singular vectors with the smallest and largest singular values. [3] This subspace actively writes high-frequency tokens into the embedding space. [1] When the projection of a reverse-engineered "average" token onto this subspace is truncated, the logits of those frequent tokens are significantly disrupted. [3] To counteract the effect, the team built EmbedFilter, a simple linear transformation that filters out the edge spectrum subspace. [1] By removing that component, the method suppresses the influence of high-frequency tokens and strengthens semantic representations. [1] The approach also produces an inherent dimensionality reduction, which lowers index storage requirements and speeds up retrieval while preserving the refined embedding quality. [1] Experiments across multiple LLM backbones showed that models equipped with EmbedFilter achieve superior zero-shot downstream performance even with significantly reduced embedding dimensions. [1] The paper's authors describe the unembedding matrix as secretly functioning as a feature lens for text embeddings and hope the findings will encourage more principled designs for training text embeddings. [9] The code for EmbedFilter has been released on GitHub. [1]