CALIBER: Calibrating Confidence Before and After Reasoning in Language Models

CALIBER (Calibration Before and After Reasoning) is a novel approach that improves confidence estimation in reasoning language models by distinguishing between pre- and post-answer confidence assessments. It reduces Expected Calibration Error (ECE) by 52.5% on the BigMathDigits dataset for a 7B model and achieves competitive results on a larger 30B model, demonstrating significant improvements in calibration, Brier score, and AUROC across various benchmarks, particularly under distribution shifts. This method is crucial for practitioners as it enhances the reliability of model outputs, especially in complex reasoning tasks, by aligning confidence estimates with the model's state of information.