LVLMs and Humans Ground Differently in Referential Communication

The article presents findings from a referential communication experiment highlighting the limitations of Large Vision-Language Models (LVLMs) in modeling common ground during interactions. The study involved human-human, human-AI, AI-human, and AI-AI pairs in a multi-turn dialogue setup, revealing that LVLMs struggle to generate and resolve referring expressions effectively. The authors released a corpus of 356 dialogues and an online data collection pipeline, which are valuable resources for researchers aiming to enhance AI's collaborative capabilities with human users.