Robusto-2: Benchmarking Humans & VLMs for Autonomous Driving in Lima & New York City

The paper introduces Robusto-2, a benchmark that evaluates the performance of Visual Language Models (VLMs) and human drivers in autonomous driving scenarios across Lima and New York City. It employs a Visual Question Answering (VQA) framework to analyze responses to dashcam footage, revealing that while humans and VLMs diverge in their answers based on question type, geographic differences do not significantly influence outcomes. This work is significant for AI practitioners as it highlights the challenges VLMs face in out-of-distribution scenarios and provides an accessible dataset for further research in autonomous driving systems.