Succeeding at Scale: Enterprise Retrieval Benchmark Construction and Index-Preserving Query Adaptation for Multi-Tenant Search

The article introduces DevRev-Search, a passage retrieval benchmark designed for technical customer support, addressing the challenges of domain adaptation in large-scale multi-tenant retrieval systems. It employs a fully automated pipeline for candidate generation using a combination of sparse and dense retrievers, enhanced by an LLM-as-a-Judge for relevance labeling. The study also presents index-preserving query-only adaptation strategies, demonstrating that fine-tuning only the query encoder significantly improves quality while maintaining efficiency, which is crucial for scalable enterprise retrieval solutions.