Rethinking RAG in Long Videos: What to Retrieve and How to Use It?

The article introduces V-RAGBench, a new benchmark designed for evaluating retrieval-augmented generation (RAG) in long, egocentric videos, addressing limitations in existing benchmarks that allow queries to be answered without video context. It also presents CARVE, a method that utilizes parallel retrievers across various configurations and employs chunk-adaptive reranking to optimize retrieval for each video segment, resulting in improved performance over eight recent VideoRAG baselines. This advancement is significant for practitioners as it enhances the accuracy of retrieval in video-based applications, enabling more effective integration of multimodal data in generative tasks.