Look Before You Zoom: Adaptive Routing for the Resolution-Context Trade-off in Visual RAG

The article introduces ViRGo (Visual Retrieval or Global Perception), a lightweight framework designed to optimize visual retrieval in Vision-Language Models (VLMs) by addressing the resolution-context trade-off. ViRGo dynamically assesses object scale using the VLM's localization heads and selects the most effective retrieval method—global perception, patch-based, or attention-based—based on the object's size, enhancing accuracy and reducing inference time across various Visual Question Answering (VQA) benchmarks. This adaptive routing approach allows practitioners to balance detail recovery for small targets with maintaining context for larger objects, improving overall model efficiency in real-time applications.