Closing the Calibration Gap in Semantic Caching

The article introduces a new evaluation framework for semantic caching in large language models (LLMs), addressing the inadequacies of the PR-AUC metric by proposing two novel metrics: Precision-Cache Hit Ratio (P-CHR) AUC and Calibration Retention Rate (CRR). These metrics account for cache utilization and the retention of ranking quality in real-world deployments, revealing that the operational gap between offline model performance and deployment efficacy is primarily influenced by the training objective rather than dataset scale. This work emphasizes the importance of calibration in model selection for semantic caching, suggesting that practitioners should focus on calibration metrics to improve deployment outcomes.