Clusters are All You Need: Pre-Training the Tsetlin Machine with Semantic Clusters from Language Models for Interpretability

A new framework proposes pre-training the Tsetlin Machine using semantic clusters derived from language models, aiming to combine interpretable reasoning with performance competitive with BERT, according to a preprint posted to arXiv on June 18, 2026 [1][2]. The Tsetlin Machine (TM) is a logic-based architecture that learns patterns using propositional clauses, offering fully transparent decision-making. However, it has historically captured limited semantic information. Pre-trained language models such as BERT achieve high accuracy on text classification but operate as black boxes, restricting their deployment in high-stakes domains [1][2]. Prior efforts to connect the two technologies relied on static word embeddings, which fail to capture contextual meaning [1][2]. The new method, detailed in the preprint, avoids embeddings entirely. Instead, text samples are grouped into semantically coherent clusters using K-means or Top2Vec. These cluster-sample pairs then pre-train a non-negated TM with enhanced Type I feedback, allowing the model to learn interpretable semantic keywords that are later fine-tuned on downstream tasks [1][2]. Across five datasets, the approach substantially outperformed vanilla and embedding-based TMs and reached performance competitive with BERT while preserving full interpretability [1][2]. The paper was submitted to the Computation and Language section of arXiv, an open-access repository that hosts preprints across physics, mathematics, computer science, and related fields. As of November 2024, arXiv receives roughly 24,000 new articles per month and has surpassed two million total submissions since its launch in 1991 [6]. The preprint’s landing page includes links to experimental community tools such as Bibliographic Explorer and Connected Papers, which are part of the arXivLabs framework launched in 2020 to enable third-party collaborators to build features on top of the repository [4][5].