TokenMinds: Pretrained User Tokens and Embeddings for User Understanding in Large Recommender Systems

A new industrial-scale system called TokenMinds generates both discrete Semantic ID-based user tokens and dense user embeddings, extending the PLUM framework from item retrieval to user modeling for large recommender systems, according to a paper submitted on 23 Jun 2026 [1]. The system adopts an encoder-decoder architecture adapted from pre-trained large language models. The decoder generates SID-based user tokens, while the encoder processes sequential user features and simultaneously produces dense user embeddings, ensuring compatibility with existing downstream models that rely on dense embeddings [2]. This dual-output design provides the complementary benefits of discrete, semantically grounded user representations while maintaining compatibility with current infrastructure [1]. Traditional user modeling in industrial recommender systems typically produces dense embeddings, which suffer from representational constraints inherent to fixed-dimensional vectors [3]. An emerging alternative using LLMs to generate text-based user tokens captures topical co-occurrences rather than deep sequential behavior dynamics and produces outputs that are difficult to ground to item attributes [4]. SID-based item tokenization has proven effective for improving generalization in generative recommendation, yet discrete SID-based representations for users remained largely unexplored until this work [1]. The shared SID vocabulary naturally extends to cross-scenario modeling. By unifying long-form and short-form video behaviors into a single model, the approach substantially reduces training and serving costs [2]. To serve the model for billions of users under strict latency constraints, TokenMinds is deployed atop an asynchronous serving infrastructure that decouples heavy representation generation from real-time scoring [3]. Validation included extensive offline experiments and live launches on multiple YouTube surfaces, served on full user traffic [1]. Focusing on ranking as the primary downstream use case, results confirm the practical viability of SID-based user tokens at industrial scale and demonstrate that tokens and dense embeddings provide complementary value across different production ranking systems [4]. Information retrieval systems, of which recommender systems are a prominent application, are designed to identify and retrieve information system resources relevant to an information need [6]. The field has seen increased attention to algorithmic bias, which describes systematic and repeatable harmful tendencies in computerized sociotechnical systems to create unfair outcomes [7]. As algorithms expand their ability to organize society and behavior, concerns have grown about how unanticipated output and manipulation of data can impact the physical world [7].