Gold Points Sniper: Self-guided Visual Reasoning in VLM for Fine-grained Action Understanding

The article introduces Gold Points Sniper (GPS), a novel framework designed to enhance lightweight vision-language models (VLMs) with self-guided multimodal reasoning for fine-grained human action understanding. It includes three modules: Gold Points Extractor for identifying critical action-relevant details, Selective Socratic Questioner for refining these details, and Semantic Entailment Evaluator for assessing factual consistency. Experimental results show that GPS significantly improves performance on a curated dataset, achieving results comparable to proprietary models like GPT-4o while ensuring higher factual accuracy, which is crucial for reliable human-robot interaction in everyday environments.