On the Stability of Prompt Ranking in Large Language Model Evaluation

This paper presents a systematic study on the stability of prompt rankings in large language models (LLMs), evaluating three open-weight models across two benchmark tasks. The authors find that while rank correlations are generally moderate to high, the top-performing prompt can vary significantly with minor changes in evaluation conditions. They propose a stability-aware selection strategy using a lower confidence bound to improve robustness in unstable settings, emphasizing the need to consider evaluation uncertainty in prompt selection and benchmarking for LLM practitioners.