Clusters are All You Need: Pre-Training the Tsetlin Machine with Semantic Clusters from Language Models for Interpretability

The article introduces a framework for pre-training the Tsetlin Machine (TM) using semantic clusters derived from pre-trained language models like BERT, aiming to enhance interpretability in text classification. By employing K-means or Top2Vec for clustering, the TM learns interpretable semantic keywords without relying on static embeddings, achieving competitive performance with BERT across five datasets while maintaining transparency. This approach is significant for practitioners as it combines the interpretability of TM with the contextual understanding of language models, making it suitable for high-stakes applications.