CalVerT: Augmenting Agents with Calibrated Verifier Telemetry Improves Action and Learning in Knowledge-Intensive Tasks

The paper introduces Calibrated Verifier Telemetry (CalVerT), which enhances LLM agents in knowledge-intensive question answering by providing a calibrated self-confidence score and a grounding verifier score. This approach mitigates failure modes such as committing to unsupported answers and unnecessary retrieval, resulting in improved F1 scores across four QA benchmarks. CalVerT can be integrated into existing QA frameworks without additional training, demonstrating significant performance gains in both training-free and training-based scenarios, which is crucial for practitioners focusing on efficient and accurate LLM deployment.