Efficient Rationale-based Retrieval: On-policy Distillation from Generative Rerankers based on JEPA

The article presents Rabtriever, a rationale-based retrieval model that utilizes an on-policy distillation framework to reduce the computational costs associated with traditional cross-encoding methods. By employing the Joint-Embedding Predictive Architecture (JEPA), Rabtriever independently encodes queries and documents while achieving comparable performance to LLM-based generative rerankers, optimizing retrieval efficiency from quadratic to linear complexity. This advancement is significant for practitioners as it enhances retrieval capabilities in applications like empathetic conversations and robotic manipulations, while maintaining strong performance on traditional benchmarks such as MS MARCO and BEIR.