Brain-IT-VQA: From Brain Signals to Answers

The article presents Brain-IT-VQA, a novel framework for visual question answering (VQA) using fMRI signals, which significantly enhances performance over prior methods by integrating the Brain Interaction Transformer (Brain-IT) to decode language tokens from brain activity. A key feature is the introduction of the NSD-VQA dataset, which offers 20 question-answer pairs per image across 20 controlled categories, facilitating more reliable evaluations of visual understanding from fMRI data. This advancement not only improves predictive capabilities but also aids in the exploration of brain representations related to visual and semantic information processing, making it relevant for practitioners working on brain-computer interfaces and cognitive AI systems.