Entity Labels Are Not Entity Signals: A Framework for Observable Relevance in Document Re-Ranking

The paper introduces a framework distinguishing between Conceptual Entity Relevance (CER) and Observable Entity Relevance (OER) in entity-aware document retrieval, arguing that relying solely on CER can lead to ineffective ranking signals. The authors demonstrate that OER provides a more reliable measure, achieving up to 10x improvement in non-relevant document pruning and a 0.051 increase in open-world Mean Average Precision (MAP) over BM25. This shift in focus from conceptual to observable relevance is critical for practitioners aiming to enhance retrieval performance in AI systems.