Mind the Heads: Topological Representation Alignment for Multimodal LLMs

The paper introduces Head-Wise Representation Alignment (HeRA), a novel technique for improving Multimodal Large Language Models (MLLMs) by enforcing alignment at the individual attention head level rather than a fixed layer. HeRA utilizes the Mutual K-Nearest Neighbor (MKNN) alignment metric and a contrastive objective to enhance cross-modal representation alignment, leading to improved performance on vision-centric tasks and reducing visual hallucinations. This method is significant for practitioners as it offers a more granular approach to multimodal training, potentially leading to better model robustness and accuracy in vision-related applications.