Latent Confidence Alignment for LLM Self-Assessment

The paper introduces Latent Confidence Alignment Error (LCAE), a novel framework for evaluating confidence calibration in large language models (LLMs) by integrating a Rasch model-based latent ability framework with item difficulty. This approach enhances the consistency between model self-assessment and the latent error probability, demonstrating improved self-assessment quality across 20 models tested on a medical-domain dataset. This advancement is significant for practitioners as it provides a more accurate measure of model confidence, which can inform better decision-making and resource allocation in AI applications.