Extraction and Analysis of Multimodal Concepts in Vision Language Models through Sparse Autoencoders

The article presents a framework utilizing Sparse Autoencoders (SAEs) to extract and analyze visual, textual, and multimodal concepts from Vision Language Models (VLMs), addressing the limitations of existing methods that treat these modalities separately. Experiments conducted on the LLaVA-NeXT VQA dataset show an improvement in visual concept quality by up to 45% compared to previous SAE-based approaches, while maintaining high quality for textual concepts. This work enhances understanding of VLMs' internal processing, facilitating better interpretation and utilization of multimodal concepts for practitioners in AI.