Listening makes Vision Clear for VLMs

The paper introduces Prompt-Vision Token Activation Map (PV-TAM), a novel approach for evaluating vision-language model (VLM) consistency by addressing issues of decoding drift and bias from structural tokens. PV-TAM enhances alignment measurement by incorporating peak attention distribution rather than solely relying on overlap masks, leading to improved performance in localization metrics across multiple datasets. This method is significant for practitioners as it provides a more reliable evaluation of VLMs, potentially leading to better model training and deployment strategies.