The Score Granularity Gap in Black-Box LLM Classification: A Comparative Study of Confidence Constructions

This study examines the "score granularity gap" in black-box LLM classification, analyzing seven methods for constructing confidence scores across 25 model-dataset pairs involving 9 LLMs. It finds that while single-shot verbalized confidence can effectively rank cases, it offers limited threshold granularity, which impacts decision-making in deployment. The research provides insights into how different confidence constructions affect model performance and inference costs, offering practical guidance for practitioners on optimizing confidence score usage in LLM applications.