When Retrieval Metrics Mislead: Measuring Policy Signal in Long-Horizon Tool-Use Agents

The study evaluates the effectiveness of exact-match retrieval recall as a measure of policy utility in long-horizon tool-use agents, specifically using Qwen2.5-3B/7B classifiers within the tau-bench framework. It demonstrates that while a compact structured state improves macro-F1 scores by 0.13-0.17, the retrieval of policy clauses does not significantly differ from gold clauses in terms of classification performance, suggesting that reliance on exact-match recall may misrepresent the utility of retrieved policies. This finding emphasizes the need for practitioners to consider integrating retrieved policies in the classification loop rather than depending solely on recall metrics for evaluating retriever performance.